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It's December, which means we can take a look back at the year and see how far 
we've come. Sys Admin has again weathered some stormy industry lows and fierce 
economic challenges. Again, we've watched sister magazines fold and witnessed 
colleagues laid off. The fact that it's December, however, also means we can put 
this difficult year behind us and look toward the year ahead. 

When budgeting and brainstorming for the coming year, the question is 
asked whether we must plan to survive or plan to succeed. I vote for the latter. 
As the economy recovers, we should all be poised to take full advantage of it. 
Operating in survival mode and simply reacting to crises may be necessary 
sometimes, but it’s also short sighted. One of the goals of this magazine is to 
take a longer view of systems administration and provide tools and information 
to help you succeed. 

To that end. a couple of specific articles in this issue provide information on 
which to build. Cameron Laird begins a series on Python with an article called 
"Python in Systems Administration: Part I — Belter Scripting". In this article, 
Laird says he finds Python "so capable and easy to learn” that he tells newcomers 
it's the best single language to choose. Within the next several issues. Laird will 
support this view by providing an introduction to the Python language, discussing 
the “scripting mentality", and presenting many practical examples of how sys 
admins can put Python to use. 

This issue's short, pull-out Solaris administration supplement features an article 
by Justin Buhler and Adrian Cockcroft that provides tips for doing capacity plan¬ 
ning. The authors introduce key principles and tools (specifically the SE Toolkit 
and Orca) and show' how to apply those principles and use the tools to obtain a 
visual representation of your servers and applications that will give you a better 
understanding of the capacity of your system. 

Sys Admin is always looking for articles that can help you tweak your sys¬ 
tems, sharpen your scripting skills, and build a plan to succeed. We're looking for 
articles on advanced administration topics, and the issue themes for the next few 
months include Security, Performance Tuning. Storage, and Networking. If you 
have tips for success in these or oilier areas, share that knowledge with your peers 
through an article in 5y.v Admin. Please send article proposals to Sys Admin's 
managing editor. Rikki Endsley at: rendsley@cmp.com. As always, you can send 
feedback and suggestions to me at: aankerhol z@cmp.com. 


Sincerely yours, 



Amber Ankerholz 
Editor in Chief 
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SCRIPTING 


Python in Systems Administration: 
Part I — Better Scripting 

Cameron Laird 


O ne well-honed item in every sys admin's toolbox is a pre¬ 
ferred scripting language — maybe sh. or ksh. or Perl, or 
something more unusual, like Rexx or Ruby. I find 
Python so capable and easy to learn that in my consulting role at 
Phased, i tel! newcomers it's the best single language they can 
learn. Python spans a wider range 
of applications than any other lan¬ 
guage I know — even more than C 
or Java — and beginners pick it up 
quickly. 

Over the next several months, 

Sys Admin's series on “Python in 
Systems Administration" will tackle 
the following four themes and their 
relations: 

• An introduction to Python 
• The “scripting mentality” 

• Practical examples of systems 
administrators' use of Python 
• The craft of systems administra¬ 
tion and its best practices 

Focus on Flexibility 

Of these, the most challenging to explain is what I call the 
“scripting mentality”, or scripting attitude. This mentality suspects 
an application that meets all nominal requirements will also be too 
“heavy” and difficult to be used effectively. For scripters, a better 
strategy for solving problems is a simple scripting interface that 
allows users to “snap together” their own solutions. The idea is to 
provide building blocks that are flexible and simple to understand. 

This idea can be a hard pill to swallow at times, because it con¬ 
flicts with the "code re-use" imperative we all learn. Tim Peters is a 
senior engineer with Zope Corporation whose standing in the 
Python community is second only to Python's inventor. Guido van 
Rossum. Peters once made the case in these words: “It's easier to 
write appropriate code from scratch in Python than to figure out 
how to use a package profligate enough to contain canned solutions 
for all common and reasonable use cases.” 

“Find'’ provides a good example of this contrast in styles. 
Windows has a “Find” application available through the standard 
“Start” button. This Find is easy to use. in that end-users regularly 
take advantage of it without documentation: point, click, and type in 
a file name, and a list of matches appears. 

The Unix find utility, on the other hand, is notorious for its 
obscurity. Few people remember all its options, and beginners fre¬ 
quently mistype even its simplest forms. The command-line utility 
scales much, much better, though. Windows “Find" makes a few 


searches very easy, but is frustrating if you need anything beyond 
the scope of its standard forms. Unix's f i nd emphasizes orthogo¬ 
nality. so you can freely combine predicates in any combination, 
even those unimagined by the designers of the utility. Once you 
learn its minimal syntax, it's a small step to request, for example, 

a list of all files with a certain 
extension, created or modified in 
the last week, which are at least a 
megabyte in size, except those 
owned by a particular user. 
Windows simply doesn't consider 
the possibility. 

The Unix find also plays nicely 
with other applications: it's easy to 
combine it with third-party filters. 
Windows, however, gives no such 
opportunity, apart from tedious cut- 
ting-and-pasting of results. 

This emphasis on flexibility 
and power will recur throughout 
this series on Python in systems 
administration. Python rarely 
gives systems administrators fin¬ 
ished solutions; it almost always 
provides highly capable and reliable tools we can use to solve spe¬ 
cific problems in just a few lines of readable code. Enthusiasm for 
such an opportunity is the “scripting mentality” this series aims to 
transmit. 

Easy Scripting 

A few examples of working code illustrate Python's advantages 
as a scripting language. Suppose you occasionally need to restore 
tiles pulled from a backup tape. From the Unix command line, you 
might request: 

tar xvf /dev/nrtape FI LEI FILE2 FILE3 

In practice, it is often useful to package up even such a simple com¬ 
mand as a one- or few-line script: if you don't work with it every 
day. you might not remember the exact name of the tape device, or 
the syntax for file names. Generalize the command above, then, into 
the shell file my_restore: 

#!/bin/sh 

# Usage: "my_restore NAME" retrieves all files in the whole 

# directory tree which match NAME: thus "my_restore my_source" 

# will find 

# my_source.c 
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# project/sources/my_source.c 

# project/sources/my_source.h 

# without requiring you to know the tree 

# structure. 

# 

# A production version of this utility 

# would be far more careful about 

# error-handling. 

DEVICE=/dev/nrtape 
LIST_OFJUALIFIEDJAMES-tar tf \ 

$D£VICE | grep $1 

tar xvf $DEVICE $LIST_0F_QUALIFIE0_NAMES 

While Python can do the same, and even 
without the minor expense of grep as an 
external process, it has no advantage over 
sh for simple pipelines: 

#! / usr/local/bin/python 

# This is my_restore.py. 

import os,sys 

device = '/dev/nrtape' 

# Invoke the application as, for 

# example, "my_restore.py MYFILE". 
pattern = sys.argv[l] 


# tar puts names on separate lines. 

# The count!) method substitutes for 

# grep. 

list_of_qualified_names = [file for file in \ 
os.popent'tar tf %s‘ % \ 
device).read!).split!’\n’) \ 
if file.coufit(pattern) > 0] 

# tar expects white-space-a single 

# blank, for example-to 

# separate its input arguments, 
files = ’ '.join(list_of_qualified_naines) 
print os.popen!'tar xf %s %$’ % \ 

(device, files).read!) 

If Python is no more succinct than sh for 
such simple tasks, why should it interest 
systems administrators? It shouldn't, 
except that our jobs never slay this simple. 
As tasks become even slightly more com¬ 
plex. two potent Python advantages 
become apparent: 

• Python's sophisticated data structures — 
including dictionaries, first-class func¬ 
tions, full-fledged objects, and more — 
are far more capable than the crude 
string-based values and variables of sh. 
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• Python truly is the "batteries-included" 
language. Once we've gone to the mini¬ 
mal trouble of writing rfiy__restore. py in 
Python, all the riches of the built-in 
Python libraries are available. The next 
section of this article shows how it takes 
only a few more lines to add a useful 
graphical user interface (GUI) to 
my_restore. It’s not just a GUI that 
Python provides, either; it's equally 
straightforward to add network services, 
sophisticated calendar arithmetic, data¬ 
base interfaces, XML processors, host 
monitors, or much, much more. 

Too often, systems administrators resign 
ourselves to the fiction that there's a 
sharp boundary between what we do. and 
"real programming". One of Python's 
great achievements is to demonstrate a 
gentle, smooth transition from few-line 
scripting to sophisticated, polished 
automation that exploits al! a platform 
offers. Add Python to your personal 
toolkit, and you immediately expand the 
range of applications you can complete 
within a demanding schedule. 

Wrap It Attractively 

Python's built-in ability to construct 
GLTs is a good example. Let's touch up 
this restore program so that a user can 
control its operation through a simple 
GUI. The soal will be to launch it, as 
before, with a simple command-line invo¬ 
cation such as my_restore.py FILENAME. 
Rather than extracting all matches for 
FILENAME, though, this version of 
my^restore. py presents a list of selec¬ 
tions. The user selects only the file 
instances he truly wants restored (see 
Figure 1). 
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Most modern Unix distributions 
include Python, and good binary and 
source versions are available from 
http://python.org/downl oad for essen¬ 
tially all platforms that appear in the 
pages of Sys Admin. 

Bring up a (shell) command line, 
therefore, and try: 

# python 

With any luck, you'll see: 

Python 2.3+ (#2. Aug 10 2003, 11:33:47) 

[GCC 3.3.1 (Debian)] on linux2 
Type "help", "copyright", "credits" or 
"license" for more information. 

>» 

Don't worry if your version is different 
from this one. The Python language is 
quite conservative; plenty of applica¬ 
tions written in the mid-'90s work 
unchanged today, and all the experi¬ 
ments described here are compatible 
with all Python releases dating back 
several years. 

You can use Python in any of a hand¬ 
ful of "modes”. Most immediately- 
rewarding is the interactive one begun by 
the command-line invocation above. 
Treat this as a “calculator”: 

>» # Lines that begin this way are comments. 

»> # When you type in what appears 
# after ">» ”, you 

>>> # should see these results: 

»> 2 + 3 
5 

»> 7’ + ’3' 

73 ’ 

»> alphabet - 'abcdefghijklmncpqrstuvwxyz' 
>» characterjist = list(alphabet) 

»> characterjist.reversed 
>» reverse.alphabet - ". joint character J f st3 
>» print neverse_alphabet 
zyxwvutsrqponmlkjihgfedcba 

Python also lets you “batch” your compu¬ 
tations. If you create a file called 
iy_scri pt.py with contents: 


2 + 3 
7' + ’3' 

alphabet •= 'abcdefghijklmnopqrstuvwxyz’ 
character_1ist = list(alphabet) 
characterjist. reversed 
reverse_alphabet = 

'’.join(characterJ1st) 
print reverse^! phabet 

you can run it “automatically” by invoca¬ 
tion at the command line of python 
my_script.py. Executing it yields the 
output you'd expect: 

5 

23 

zyxwvutsrqponmlkjihgfedcba 

As we pursue our work with Python, 
we’ll come across at least a couple of 
other execution “modes”. Interactive 
and batch interpretation are much the 
most important for your start. Even if 
you are accustomed to work with com¬ 
mand-line BASIC or Unix shells, 
though, it’s likely to take a while to 
appreciate the richness of Python’s 
interaction. 

Python embeds powerful document¬ 
ing and introspective capabilities that 
make it inviting, for example, to “ask the 
language” itself how common tasks are 
done. If you’re fuzzy on how Python lists 
work, start by printing out the inter¬ 
preter’s own help on the subject; 

# python 

Python 2.2.1 (#1. Aug 30 2002, 12:15:301 
[GCC 3.2 20020822 (Red Hat Linux 
Rawhide 3.2-4)] on linux2 
Type "help", "copyright", "credits" or 
"license" for more information. 

»> help(list) 

Help on class list in module bui1tin : 

class 1ist(object) 
j listd -> new list 
j 1 i st( sequence) -> new list \ 
initialized from sequence's items 

I 

| Methods defined here: ... 

Future articles will give more practice in 
taking advantage of Python’s self-docu¬ 
menting benefits. 
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Python requires only a few additional lines to implement such a 
GUI: 

#!/usr/local/bin/python 

# This is ’my_restore.py’. Invoke it as, for example, 
if my^restore.py my_source.c 

# to show all the archived files named 'my_source.c' 

# which the user can select for retrieval. 

import os.sysJkinter 

pattern = sys.argv[l] 
archive = '/dev/nrtape' 

def esft): 

# Construct the string which lists all selected files, 

§ separated by a single blank. Use that string to 
if specify the exact listof files to extract from the 
if archive. 

command = 'tar xf SJs %s' % (archive, 

’ \join([lb.get(index) for index in Ib.curselectionO])) 
os.systemt command) 

if MULTIPLE so that we can select and extract several files in a 
if single operation. 

lb - Tkinter.Listbox(height - 12, width = 30, selectmode = \ 

Tkinter.MULTIPLE) 
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Ib.packf) 

Tkinter.Buttonttext = "Extract selected files", command = \ 
esf).pack() 

# The says, "ignore the trailing newline tar emits”, 

for qualified_name in \ 

os.popen{’tar tf Xs’ % archive).readt).split( 1 \n')[:-1]: 

# Does the basgname of this item match the pattern? 
if os.path.basename(qualified_name).count(pattern) > 0: 
lb.insert(Tkinter.END, quaTifiedjiame) 

if Show the GUI panel. 

Tkinter.mainloop{) 

This example dramatizes the contrast with sh and other conven¬ 
tional administrative scripting languages. While a few lines in 
each suffice for simple tasks, requiring a GUI or other common 
features quickly outstrips the ability of the usual shell languages. 
If you're coding in Pvthon. though, a solution demands only a few 
more lines. 

Here’s the strategy, then — adopt Python as one of your pre¬ 
ferred scripting languages. Use it for the day-to-day chores that 
wise sys admins solve by scripting. When you return to the same 
task in eight months, you'll find your well-written Python source so 
readable that you don’t need ancillary READMEs or instruction 
sheets or recipes. Moreover, when you need more serious develop¬ 
ment {e.g.. a GUI for a little task, or service of remote requests). 
Python will be ready to express your new version with modest 
incremental effort. 

Over the coming months. I'll compose several scripts that solve 
specific problems that often arise in systems administration. Future 
installments will also focus on how Python not only makes essential 
things easier, but also makes important things possible. 

In the meantime, you can find more information on the topics 
mentioned in this article at these sites: 

Tutorial on find — http://www.grymoire.com/Unix/Find.litiiil 
The original christening of Python as the "batteries included" 
language — http://web.archive.org/web/20001013152452/ \ 
http://sunworld.com/swol-12-1998/swol-12-regex.html 
“Python is an Agile programming language” — 
http://www.oreillynet.com/pub/wlg/3060 

Cameron Laird, a vice president at consultancy Phase it. Inc. 
http:/Zphaseit.net/. has written occasionally for Sys Admin in previous 
years. 


ATTENTION SUBSCRIBERS 

Renew your 
subscription 
online & save $! 

www.Sysadmirimag.com/sub/ 
Discount Keycode: 2SRW 



10 — Sys Admin 


www.sysadminmag.com 


December 2003 















TOOLS 


Tepatche — Automatic Open BSD 
System Patcher 

Gunnar Wolf 


O penBSD is my preferred operating system for highly 
exposed or security-oriented servers. OpenBSD is designed 
specifically with security in mind and. with “only one 
remote hole in the default install, in more than 7 years" {according 
to www.openbsd.org) it becomes the obvious choice for any 
exposed server. 

No code written by humans, however, can be considered perfect, 
and OpenBSD is no exception. A new OpenBSD release comes out 
every six months, and some flaws are always found. Because 
OpenBSD is a security-minded system, the core team releases a 
source code patch every time a bug is found and corrected. During 
the life of any given release, an average of 30 patches are issued (see 
sidebar “OpenBSD Releases' Life Cycle"). 

Patching an OpenBSD System the 
Standard Way 

OpenBSD w'as designed with the idea that a simple implemen¬ 
tation lends to be and stay more secure than a complex one. 
Llpdates are distributed as source code patches. To install them, 
the svs admin must patch the OpenBSD sources and recompile 
and install the relevant portions of 
the system. This process is easy to 
implement since patches come 
with complete instructions. 

Keeping the systems current can be 
done by subscribing to security- 
announced openbsd.org. down¬ 
loading the patches, then applying 
and compiling them. Keep in mind, 
however, that: 

I If you manage many OpenBSD 
machines, this process can be 
tedious. Applying the patches to a 
couple of machines is easy, but 
patching a whole data center can 
take hours, particularly if you 
check the status of each machine 
(as you should always do). 

2. Patching may be postponed for days or weeks, or even forgotten 
despite our best intentions. This is especially a problem because 
automatic worms are scanning networks and compromising vul¬ 
nerable systems. 

Therefore, I wrote Tepatche to address these problems. I wrote the 
first version in June 2002. shortly after the first (and onlv) known 



remote vulnerability in the default configuration of OpenBSD sur¬ 
faced. Some blaekhat groups promptly released exploits for this 
vulnerability, and many systems administrators were left vulnera¬ 
ble. I decided to write Tepatche because periodically checking and 
applying simple patches is a task that a computer can do better than 
a human. See the sidebar “Why ‘Tepatche ?". 

Program Logic 

Tepatche should be inn automatically; l suggest running it daily 
as a cron job. Its operating logic is quite simple. Roughly: 

• Tepatche reads the address of the FTP site for updates (almost 
always ftp.openbsd.org/pub/OpenBSD/patches/<vers 1 on>/j 
from the configuration file and initiates a connection to the site. 

• It obtains the list of files available in both the common and archi¬ 
tecture-specific directories of the FTP site, compares it to our 
local copy, and if there are any new files, retrieves them, and 
schedules them for processing. 

• It processes the patch instructions, executes each patch instruc¬ 
tion. then checks whether it is in the allowed set of instructions. 
(This is explained in greater detail in the next section.) 

• Tepatche notifies the administra¬ 
tor of any changes made by send¬ 
ing two emails —- one with the 
full output of the execution, and 
one w ith a short summary. 

The FTP connection is handled 
using Perl module Net:;FTP, which 
can be found either in the CPAN or 
as p5-libnet in OpenBSD’s ports 
tree. 


Important Design 
Considerations 

As I mentioned, the program 
logic is simple. The whole program 
is slightly more than 400 lines of 
code, about 23% of which is com¬ 
ments. However, this program is intended to run as root because it 
has a lot of interaction with the operating system and is intended to 
modify vital system binaries. 

1 did my best to ensure that no error condition would go unno¬ 
ticed. checking almost every statement that uses the FTP module, 
invokes an external command, or deals with the tile system. This 
resulted in almost 60 possible causes for early program termination. 
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Of course, early termination is always promptly reported to the 
systems administrator. 

Because OpenBSD patches are distributed by FTP and they are 
not cryptographically signed, we can never be sure whether they 
were issued by a real OpenBSD developer or they are trojans, so we 
must trust that the tile is legitimate for this process to be automated. 
Because the patches’ headers are basically recipes on how they are 
to be applied. I decided to deal with this trust by being very selec¬ 
tive about which commands Tepatche will automatically execute. 
The commands Tepatche will agree to run are: 

• cd — You must always change to the affected directory before 
patching. 

• patch— The command that applies the patch. 

• make — Compile and install the affected portion of the code. 

This is enough for the vast majority of the patches, but not for all of 
them. One third of the patches in 3.2 involve killing a running 
process or removing older files. Although it is currently not imple¬ 
mented, I intend to include a configuration option (disabled by 
default) that allows Tepatche to execute any command listed on the 
patches' headers. 

Finally, although most patches are applied to userland (i.c.. the 
binaries, libraries, and other tiles available in a Unix environment), 
some patches must be applied to the kernel. I decided Tepatche 
would only patch the kernel source tree but not rebuild the kernel. 
Many people custom-con figure their kernels, and it's not easy to 
come up with a standard version of compiling the kernel. The only 



Tepache is a popular, slightly alcoholic drink in Mexico 
( where I live and where this program was created). Tepache is 
the result of fermenting pineapple in water. Quoting from 

http://www.fao.org/docrep/x0560e/x0560e09.htm: 

Tepache is a light, refreshing beverage prepared and consumed 
throughout Mexico. In the past, tepache was prepared from 
maize, but nowadays various fruits such as pineapple, apple and 
orange are used. The pulp and juice of the fruit are allowed to 
ferment for one or two days in water with some added brown 
sugar. The mixture is contained in a lidless wooden barrel called 
a "tepachera". which is covered with cheesecloth. After a day or 
two. the tepache is a sweet and refreshing beverage. If fermenta¬ 
tion is allowed to proceed longer, it turns into an alcoholic bever¬ 
age and later into vinegar. The microorganisms associated with 
the product include Bacillus subtil is. B. graveolus and the yeasts. 
Torulopsis insconspicna, Saccharomyces cerevisiae and Candida 
queretana (Aidoo, 1986). 

One of the best ways for a systems administrator to get enough 
free time to go and enjoy some tepache is. precisely, by having 
their patching work automated. 

If you are curious, you can find recipes to prepare tepache 
(in Spanish) at: 

http://www.chi.itesm.mx/chihuahua/arte„cultura/cocina/ \ 
behidas/tepache.html and 
http://mexico.udg.fnx/cocina/bebidas/tepache.html 



OpenBSD is a non-commercial project, led completely by 
volunteers. They follow a strict schedule of releasing two ver¬ 
sions every year —May and November. 


Many commercial companies continue to support older 
versions of their products. In OpenBSD. this is simply not 
possible —- it takes up too much of the developers’ time, which 
can be much better used at hacking at new features or the 
ongoing security audit. The OpenBSD developers will always 
support two releases of OpenBSD — the latest stable version 
and the immediately previous version. After one year, old versions 
become deprecated and are not maintained. 

Because OpenBSD is free software, this policy should not be 
much of a problem for administrators. It guarantees the users 
that OpenBSD will continue to be a high-quality, innovative, 
functional, secure, and free operating system. 


way to apply the patches is to reboot the system, and no mission- 
critical system should automatically reboot itself, even more when 
faced with the danger of having an incorrectly built, unbootable 
kernel. Thus, the administrator must always rebuild the kernel, and 
Tepatche wall only alert him that he must do so. 

Conclusion 

I have been running Tepatche for more than a year, and although 
I do not administer many OpenBSD boxes, it has clearly improved 
my life as a systems administrator. Knowing I am up-to-date with the 
patches and knowing that most patches will be automatically applied 
(or at least, 1 w ill get a notice from my computer telling me what to 
do) has made my life much easier. 1 have worked with Tepatche on 
OpenBSD versions 3.1 and 3.2 and 3.3. and have had no problems 
with it. Tepatche is free software, under the BSD license. You can 
download it from http://www.gwolf.cx/soft/tepatche/. See the 
sidebar “Similar Projects’’ for more information. 

Giftinar [Volf has been a systems administrator since 1993, with a special 
interest in security. He has worked with and promoted free software since 
1997 and is active in Mexico's free software comm units. 


I know of two projects with goals similar to Tepatche’s. but with 
different approaches. One project is Gerardo Santana Gomez’s 
binary patches project (http:// www.openbsd.org.mx/-santana/ \ 
binpatch.html). It’s not always practical to compile from 
source to keep a running system current (e.g., if a server has too 
much processor load or too little hard disk space, or if we have 
many identical machines), so distributing precompiled patches 
makes more sense than recompiling everywhere. 

Leonardo Costa wrote QpenBechede ( http: //openbechede. \ 
SOUrceforge.net/en/). The goal of OpenBechede is to manage 
OpenBSD s ports infrastructure, handling dependency chains 
and updating them. The ports are not maintained together with 
the core OpenBSD system. For a better explanation on the ports, 
refer to http://www.openbsd.org/ports.html. 


Similar Projects 
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Deploying Modules 

Dale Southard 


U nder most current Unix/XI 1 systems, the user experience is 
primarily controlled by four groups of settings: 

• The user's choice of shell interpreter tesh. sh. bash, ...). 

• The setting of environment variables like SPATH, SMANPATH. 

S1NFOPATH, SLD_LIBRARY_PATH. and SLM_LICENSEFILH. 
• Shell aliases and macros. 

•The XI l resources available through the XI 1 file search path or 
through xrdb. 

Typically, these settings are statically configured by sourcing "rc" 
files located in system directories like /etc, or in the user's home 
directory when the user either logs in or launches a shell. For simple 
environments, this system works well in both standalone systems 
and moderate-sized networks. 

Unfortunately, in larger, more complex environments that must 
serve a broader spectrum of users, a single statically configured 
environment can present problems for both users and systems 
administrators. Specifically, the following problems effect most 
large statically configured environments to varying degrees: 

1. Applications tire not "version controlled" — Users are provided a sin¬ 
gle version of most common apps. If the app is upgraded, the newer 
version is both deployed (made 
available for use) and committed 
(made the default version for all 
users) at the same time. This means 
that updates generally require 
extensive testing prior to deploy¬ 
ment to ensure that they do not 
adversely affect any existing users. 

2. Central points of failure — All 
applications maintainers require 
modification privileges on the net¬ 
work "bin” directories and rc files 
in order to make their apps avail¬ 
able to users. The larger number 
of people working with the same 
directories and files increases 
both the opportunity for making 
mistakes and the effects of those 
mistakes on the user community. For example, a single error in a 
centralized rc file can interrupt things for all users, even those whose 
work-flow was otherwise independent of the application maintained 
by the person who made the error. 

3. Namespace issues — Software packages often contain applica¬ 
tions that conflict with the names of applications in other pack¬ 
ages. This can become confusing for users, since resolving the 
conflict often means renaming an application in ways that do not 


match the documentation. Worse yet. it is easy for application 
maintainers to overwrite each other's executables or shell vari¬ 
ables when deploying new packages. 

4. Shell dependence — On well-configured systems, users are free 
to choose their shell interpreter based on personal preference. 
Unfortunately, many application maintainers. and several com¬ 
mercial applications, only support their application under a single 
shell or family of shells. 

The modules package provides an answer to these problems by pro¬ 
viding users with a dynamically reconfigurable environment that is 
independent of their choice of shells. Rather than installing all appli¬ 
cations in a common space managed by a single rc file, the use of 
modules encourages installation of each version of each application 
in its own directory tree (see the sidebar "Installing Applications in 
Custom Locations"). It encapsulates the environment changes for 
each version of each application in a separate modulefile that the user 
can load, unload, or query using the inodul e command. 

The modules package itself is open source and available via the 
Internet. It was originally described in John L. Furlani’s paper (LISA 
Conference, 1991) as a set of functions for esh and sh shells, but the 
modules package has become a standalone application with an embed¬ 
ded Tcl/TclX scripting language and now supports most common 
shell interpreters including esh, tesh. sh, ksh, bash, zsh. and Perl. 

Additional features include enhanced 
logging, modulefile tracing, and sup¬ 
port of hierarchical modulefile orga¬ 
nization. Recent releases also include 
more advanced configuration options 
and the ability to do apropos-style 
searches of the installed modulefiles. 

This article will focus on the 
basics of using and configuring mod¬ 
ules from the viewpoint of users, sys¬ 
tems administrators, and application 
maintainers. The examples used in 
this article are from a system running 
modules 2.2.2.4 under IRIX, but 
were also tested against release 3.1.6 
under Debian Linux. In the exam¬ 
ples, the module tiles were installed 
in a directory tree located at 
/depot/moduletiles, but sites could easily use other locations. 

Modules for Users 

The core of the modules system is the module command inter¬ 
preter. During shell initialization, a shell alias or macro that calls the 
module command is defined. When this macro is called, the module 
interpreter runs the specified modules sub-command, which in turn 
performs actions based on one or more modulefiles. 
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From a user's prospective, the most important function of mod¬ 
ules can be found in four of the modules sub-commands: 


myhostS nodule swap MIPSpro MIPSpro/7.3.1.2 
Switching ’MIPSpro/7.3’ to ’MIPSpro/7.3.1.2‘...ok. 


•The module list sub-command shows which modules are cur¬ 
rently loaded in the user’s shell. 

• The module load and module unload sub-commands are used to 
make a software package available for use. or to remove a package 
and its settings from the user's environment. 

• The module swap sub-command is equivalent to a mod u 1 e unload 
followed by a trod u 1 e 1 oad. It is typically used to switch between 
different versions of the same package. 

•The module avail sub-command lists all the modulefties avail¬ 
able tor the user to load/unioad/swap. This provides a mechanism 
for users to “discover” newly upgraded/installed software pack¬ 
ages without resorting to email/Web/paper memos or lists. 

The following transcript illustrates use of the avail, list, and swap 

modules commands in a simplified modules setup: 

myhost% module avail 


/depot/modulefiles 


MIPSpro 

MIPSpro/7.2.1.3 
MIPSpro/7.3 
MIPSpro/7.3.1.1 
MIPSpro/7.3.1.2 
MIPSpro/7.3.1.3 
modules 

modules/2.2.2.4 


purify 

purify/2002.05.00 
totalview 
totalview/4,1.0-1 
total view/4.1.0-6 
totalview/5.0.0-1 
total view/5.0.0-4 


myhost% module list 
Currently Loaded Modulefi1es: 

1) modules 2) MIPSpro/7.3 

myhost% cc -version 
MIPSpro Compilers: Version 7.30 


Installing Applications in 
Custom Locations 


The modules package depends upon having each version of 
each application installed in a separate directory. Though this 
may sound daunting, it is surprisingly easy to accomplish. 

Much of the available open source software is managed 
under the GNU autoconf system. The autoconf configure com¬ 
mand accepts a - - pref 1X= flag that can be used to set the top of 
the tree that the software will install into, so a command like 

configure --prefix—/depot/emacs/xemacs- 21.4.13 wall do 

the right thing for modules. 

Likewise, many public domain and commercial apps are 
installed by makefiles or shell scripts that can be modified to place 
the software in a unique directory. In many cases, a flag is provided 
for targeting the install to a non-default location (for instance the 
- r flag in IRIX’s inst/swmgr package management system). 

As a last resort, a strategically placed symlink can often be 
used to re-direct the software to another location (for example, 
Alias|Wavefront packages will go wherever/usr/aw points them). 


myhostS cc -version 

MIPSpro Compilers: Version 7.3.1.2m 

The modules package also provides mechanisms for users to incorpo¬ 
rate their own collections of modulcfiles as well as set up the default 
modules they load at login and display what changes a modulefile will 
make to their enviroment. The newest release also allows searching 
though a whati S-style database for related modulefiles. These 
advanced features are beyond ihe scope of this introductory article. 

Modules Initialization 

Installing modules for use on a system requires initializing 
modules during startup of the user’s shell and providing one or 
more modulefiles for loading/unloading/swapping. Initialization of 
modules is done by adding a stanza like the following to the appro¬ 
priate eshre or profile file: 

# 

# setup esh for modules and load/unload some modulefiles 

# 

if (-r /depot/modules/modules/fnit/esh) then 
source /depot/modules/modules/init/esh 
module load modules 
module load default 
module unload totalview 

endif 

When deploying modules, it is important to consider where mod¬ 
ules will be initialized. Two obvious choices are the centralized rc 
file for each shell (e.g.. /etc/profile for sh), or the user’s home 
directory rc files (e.g., -/.eshre for esh). 

Using the centralized rc files is somewhat easier to implement. 
Using the centralized rc files also provide a nice hierarchy for cus¬ 
tomizing the modulefiles that users are given by default: 

1. A network-wide “default" modulefile contains the modules sub¬ 
commands to load a standard set of modulefiles. 

2. Each system can modify this default set of modulefiles via the 
addition of modules sub-commands to the same rc files where 
modules is initialized (e.g.. in the above example the “totalview" 
application is unloaded after default modulefile is loaded). 

3. Individual users can add module sub commands to their startup files 
to further customize the starting set of modulefiles in their account. 

Unfortunately, using the centralized rc files for modules initialization 
comes with one huge disadvantage — the centralized rc files are not 
sourced when a non-login shell is executed. That means that sub¬ 
shells launched from within applications like vi or emacs will not 
initialize modules and thus will not allow users to utilize modules 
commands. Even more confusing for advanced users, launching 
remote applications via rsh/ssh will also fail to initialize modules. 

Finally, there are some module sub-commands like i ni tddd and 
initrm. which will not work correctly unless the modules initializa¬ 
tion is present in the user’s home directory re files. All these prob¬ 
lems are not present if the modules system is initialized from the rc 
files in each user’s home directory. 

Note that it is possible (though not recommended) to initialize 
modules in both the centralized and home directory rc files. In fact. 
I’ve used such a setup for several years without problems. 
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Modules Modulefiles 

The modules system uses one module file for each version of 
each software package that will he managed by modules. The mod¬ 
ulefiles themselves contain the commands necessary for making an 
application available to users, including modifying the $PATH or 
other environment variables, setting aliases, and making resources 
available to XII. In version 2 and later, these modulefiles are writ¬ 
ten in the Tel scripting language and begin with the string 
"#%Module”. The modules system has provided several extensions 
to Tel that are specific to managing user environments: 


# 

##### standard defs ###### 
set sys [unane sysname] 

set os [unane release] 

set mach [unane machine] 

#### program specific stuff ##### 

set root "/depot/enacs/xemacs-21.4.13" 

prepend-path PATH Iroot/bin 

prepend-path MANPATH $ root/man 

prepend-path INFOPATH $root/lib/xemacs/info 


setenv — Sets an environment variable in the shell, 
prepend*path — Adds a value at the beginning of a colon-sepa¬ 
rated list. 

append-path — Adds a value at the end of a colon-separated list, 
set-alias — Creates an alias using sh-style args. 
x-resoLtrce — Merges a resource into the XII resource db. 
module — Allows a module tile to execute other modules sub-com¬ 
mands. 

uname — Provides access to host info on the target system. 

The first four commands perform an inverse of their norma! function 
when called by a module remove sub-command. Most modulefiles 
contain fewer than a dozen lines of code. For example, the module tile 
below would suffice for controlling an installation of XEmacs: 

ftModule 

# 

# xemacs 21.4.13 modulefile 


The modulefiles are usually gathered into directory trees that are 
grouped by application. Within these directories, the modulefiles are 
generally named using the version of the application. For example: 

/depot/modulefiles/total view: 

4.1.0-1 
4.1.0-6 
5.0.0-1 
5.0.0-4 

/depot/modulefil es/MIPSpro: 

7.2.1.3 

7.3 

7.3.1.1 

7.3.1.2 

7 . 3 . 1 . 3 

With a structure like this, modulefiles can be referred to as 
“application/version". For example, the command module load 
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total view/ 5.0.0-1 would load the modulefile at /depot/module- 
files/total view/5.0.0-1 /. 

If the modulefile is not fully qualified (e.g., module load 
total view), modules will load the lexicographically highest module- 
file name (in our example, /depot/modulefiles/totalview/5.0.0-4). If 
desired, a “.version” file can be used to override the lexicographical 
soil. For example, if the following .version file were in the /depot/mod- 
ulefiles/totalview directory listed above, the command module load 
total view would default to the total view/4. 1.0-6 modulefile rather 
than the lexicographically higher total view/5.0.0-4 modulefile: 

//^Module 

set ModulesVersion 4.1.0-6 

Note that the ModulesVersion variable refers to the version file that 
should be searched for in the hierarchy rooted in the directory where 
the .version tile was encountered, not the version of modules that 
the user is running. 

Modules in Practice 

Once configured, the modules system provides a mechanism to 
address the issues mentioned in the introduction to this article: 

1. Applications are “version controlled". Users can switch between 
installed versions on the fly. In practice, this vastly simplifies the 
process of deploying new software for sys admins. New packages 
are simply installed and a new modulefile is created. “Friendly 
users" can test the cutting edge versions at their leisure while 
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those in the trenches can continue to access older versions as Ions 
as needed. No one is ever tied to a "default” version. 

2. Application maintained no longer need modification privileges 
on common directories and rc files in order to maintain soft¬ 
ware. Instead, they are given two directories: a directory in 
which to install their software releases (e.g.. /depot/xemacs). 
and a directory in which to install their modulefiles (e.g.. 
/depot/modulefiles/xemacs). Since these directories are not 
shared with other maintainers, the impact of their activities is 
limited to the applications they are maintaining. Other software 
packages are unaffected by any errors that may occur. 

3. Namespace issues arc under the control of the user. Because users can 
control which modules are loaded and in what order, name conflicts 
are either eliminated entirely or can be unambiguously controlled. 

4. Modules are shell-independent. A single modulefile will work for 
csh. tesh. sh, ksh. bash, zsh. and Perl users. 

Beyond solving these problems, the modules package also provides 
a more powerful environment for users and developers. Here are 
some examples of setups I've used: 

• Providing the environment for the ubiquitous /usr/local tree us a 
modulefile allows developers to module unload it prior to com¬ 
piling software that shouldn't be linked to non-system libraries. 

• Providing a GNU modules system that prepends the common 
GNU commands to the head of the user's environment. This pro¬ 
vides an easy route for GNU/Linux users to experience a familiar 
environment even when they are working on a non-GNU system 
like Solaris or AIX. 

• Providing “meta modulefiles” that use multiple module commands 
to load customized environments for different tasks like develop¬ 
ment, graphic design, quantum chemistry, or CAD/CAM work. 

• Providing multiple, highly customized versions of the same pack¬ 
age to better meet the needs of specific researchers or groups. 
This is a common need in quantum chemisty and other disciplines 
where researchers need specialized versions of common software 
to handle the particulars of their project. 

Additionally, the modules system forms the underpinnings of the 
OSCAR Linux clustering effort’s “env-switcher” application. The 
combination of modules plus env-switcher allows users to access all 
the power of the underlying modules system for on-the-fly environ¬ 
ment changes, as well as providing a mechanism for user-controlled 
environment changes that are persistent across logins and ssh/rsh 
remote program invocations. 

From niv experience, the single largest benefit of modules is the 
time the system saves. Because modulefiles allow deploying appli¬ 
cations without committing the entire user base to using them, tasks 
that previously required extensive planning, testing, and administra¬ 
tive overhead to prevent versioning conflicts can instead be solved 
immediately by simply installing the software in parallel under 
modules control. This is appreciated by the users as w ell as the sys¬ 
tems administrators and application maintainers. 
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BACKUP 


Oracle Database Backups via 
UFS Snapshots and Enterprise 
Backup Products 

Rick Moakley 


I n the fall of 2002, I was assigned the task of putting together a 
Solaris 9 system for a new machine, including system and data¬ 
base backups. Typically, we would install an ATL and backup 
product for this task, but we wanted to move away from that model 
to something more enterprise-centric. We use IBM’s Tivoli Storage 
Manager for our enterprise backup product, so we looked into their 
Data Protection for Oracle add-on. In the meantime, I put together 
an interim solution, as described in this article, which worked so 
well that we kept it, 

I faced the task of backing up this 250+ GB Oracle database over a 
100 Base T NIC, and the problem was the amount of time it would 
take to do so. Rough estimates indicated that it would take 8 to 12 
hours on a good day. Then, my research uncovered UFS snapshots. 
Using UFS snapshots (and a little multitasking), we w'ere able to com¬ 
plete the backup in about 4 hours (during off-shift time). With this 
setup, there is only a minute or two of downtime for the database when 
doing a cold backup, and the database is in backup mode only briefly 
during a hot backup. 

The advantages of the resulting scripts using snapshots and multi¬ 
threading the process were significant. We saw significant cost savings 
by not purchasing an ATL, tapes, and software. The downtime for a 
backup is brief, and the wall clock time is a fraction of a regular backup. 
Also, we don’t have to maintain lists tiles to be backed up — a big plus 
to any admin. Once the space is set aside for doing snapshots, the 
scripts are installed and cron’ed. and the Oracle userid is profile veri¬ 
fied, the process runs automatically and is virtually maintenance free. 

UFS Snapshots and Wrapper Scripts 

UFS snapshots are a little-used facility incorporated into Solaris 
(since release 8) that allow a privileged user (i.e., root) to create a 
point in time copy of a file system afer quieting it. Copy-on-write 
operations place pre-change copies of the filesystem blocks into a 
“backing store file", and when the associated fssnap device is 
mounted, the blocks stored in the backing store file overlay the 
changes to the original. Thus, reading from the snapshot device pre¬ 
sents a point-in-time copy of the original file system to the backup 
product of choice. 

Doing a database backup is typically more complicated than a 
file system backup, and because I was adding snapshots and multi¬ 
tasking to the mix, things got even more complicated. My solution 
was to put a couple of wrapper scripts together to manage the 
backup process and do the actual hackups. There is a backup.parent 
script (Listing 1) that does the management. It queries the database 
for its critical file information (i.e., tablespace, control filenames, 
archive logs, etc.), readies the database (i.e., puts it in backup 
mode), snapshots the required filesystems, and then starts a set of 


backup.child scripts (Listing 2) that do the actual backups. Each 
invocation of a child script is passed a filesystem, set of files within, 
and proceeds to do its share of the work backing up the tablespace 
files and deleting the snapshot when complete. 

For cold backups, the datafiles, controlfiles, and logfiles are 
backed up. For hot backups, the datafiles, archive logs in use 
between the start and end (signified by a “switch log”), and a 
backup copy of the control file is created and backed up. 

How the Scripts Work 

The scripts are pretty straightforward. The use of functions 
(although perhaps easier in some sense) was discouraged to facilitate 
easier reading by non-programmers. There are very few' variables to 
adjust, and those are detailed in this article. When started, the scripts 
are passed a database instance name (the ORACLE_SID), and 
optionally the type of backup (cold or hot), number of concurrent 
streams, and a retention period (used by TSM ): 

./backup.parent database!)! HOT 4 ARC5WEEKS 

The defaults are cold — one per CPU — and ARC5WEEKS (which 
must be defined by the TSM administrator). The parent script, after 
profiling its environment, ensures that the snapshot filesystem (i.e., 
snapshot_fWfssnap) is actually mounted (it will alert you to this 
and point out where to change this value). At one point in my test¬ 
ing, I discovered that the snapshot filesystem wasn’t mounted, and 
the process continued merrily along its way filling up root, so I 
added the above section into the script. 

Provided the snapshot_fs is okay, the next step is to determine 
which files are actually going to be backed up. I chose to avoid man¬ 
ually keeping track of files or directories, which is error-prone at best, 
and at worst you don’t know what was forgotten in a backup until you 
need to restore it. The script goes right to the source — the database 
itself — for each run. When the DBA adds a file, it doesn’t get 
missed. Files are identified by doing a “switch user” to the Oracle id 
and executing a sqlplus command with a few' selects, while spooling 
the output to a file. If your Oracle userid is not “oracle”, change it in 
the script (i.e., oracle=“oracle”): 

C0NTR0LFILE='v\$controlf11e‘ 

LOGFILE-'v\$logfile' 

DATAFILE='v\$datafile' 

TEMPFI LE=’ vUtempf i 1 e' 

Ilsu] - t[oracle} « EOF » $1 ogl 2>M 
i{prep_script) 
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sqlplus -SILENT "/ as sysdba" << EON » Slog? 

whenever sqlerror exit 1 

set pagesize 0 

set heading off 

set timing off 

spool $tmpfilel 

select 'CQNTROLFILE::'j | name from $CONTROLFILE; 
select 'TABLESPACE:’||tablespace_name||||file_name from \ 
sys.dba_data_files; 

select 1 LOGFILE::’11member from SLOGFILE; 

select ’TEMPFILE:: 1 ||NAME from JTEMPFILE; 

spool off 

set echo off 

spool Jtmpfile2 

SHOW parameter db_block_si 2 e 

spool off 

exit 0 

EON 

EOF 

The resulting file can be greped for the needed Files. When doing a 
cold backup, the control Files, tablespaces files, and log tiles must 
be backed up via snapshot. For a hot backup, we only back up the 
tablespace Files via snapshot. Special considerations are made for 
the control file and archive logs. 


Figure 1 Typical backing store sizes 
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Figure 2 Maximum backing store sizes 



Since each Filesystem must be snapped only once, we use g rep to 
select the ones we want and to weed out the ones we don't want. 
According to the above rules, run a df on each database tile, writing 
the Filesystem name to a temporary tile. A sort -u command 
removes the duplicates, resulting in a list of filesystems to snapshot: 

if [ $hot_or_cold — "HOT" ]; then 

path!i st-$( grep ^"TABLESPACE :” ttmpfilei | nawk -F: ’{print INF} 1 ) 
else pathli st=$ (grep -v "TEMPFILE:" ttmpfilei | nawk -F: ’{print $NF}’) 
fi 

for path in tpathlist; do 

filesys=$(df $ path | cut -fl -d’C|tr -d ’ ’) 
outfi1e-$(echo $filesys|tn / _) 
echo tpath >> Jtmpname.loutfile 
echo tfilesys ttmpname.loutfile >> ttmpfi1e2 

done 

# sont and de-dup the temp file 
sort -u -o $fiTesystems ttmpfi1e2 
${rm) Stmpfi1e2 

Before we snapshot the filesystems, the database must be “readied for 
backup' - . In the case of a cold backup, we do another switch user to 
Oracle, and a “shutdown immediate”. All the commands are in stream 
with another “here is” document, as shown in the previous example. 
The process is a bit different for a hot backup; we need to put each 
tablespace in “backup mode”. Because the list of files isn’t hard¬ 
coded, we use awk to build a couple of "command files” with the stan¬ 
dard commands, a complete set of BEGIN and END BACKUP 
tablespace commands, as well as a couple of commands to print the 
status of the archive logs. Note that for this method to work, the DBA 
must ensure that the database is running in “archive log” mode. 

Once we’ve prepared the database, we run through the list of 
filesystems, taking a snapshot of each one. Things get sticky if any 
of the snapshots fail. You can’t just quit and leave the database in 
backup mode, or W'orse, down. Users and DBAs don’t like that, and 
you can’t just leave the snapshots hanging there, I coded the script 
so that if a snapshot error is encountered, it stops trying to take the 
others, deletes the ones already taken, and readies the database for 
normal operations. This rarely happens, but it’s best to be prepared: 

chunksize=$(expr Sblksize / 1024Jk 
while read filesystem pathnames 
do 

outfile=$Cecho $fiTesystern11r / _) 
snapshot-$($[fssnap] -F ufs \ 

-o backing*store=$bstmpname.$outfile.bs.chunksize-tchunksize \ 

$f11esystem) 

rc=$? 

if [ Src -ne 0 ]; then 

logger -ip locall.notice $0": Error" $rc "doing snapshot \ 
for:" Ifi 1esystem 

echo "Error" $rc "doing snapshot for:" $fi 1esystem 
# error? don’t exit here {database status?) see below 
break 

else echo $fi1esystem "snapshot taken. Ok" 

logger -ip 1ocal1.notice 10":" $f11esystem "snapshot \ 
taken, Ok" 
fi 

echo $fi1esystem fpathnames $snapshot >> Jtmpfile? 
done < $fi1esystems 
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if [ $rc -ne 0 ]; then 

if [ -f $fi' 1 esystems ]; then 

while read filesystem pathnames snapshot 
do 

logger -ip local 1.notice 10": Removing snapshot" tsnapshot 
echo "Removing snapshot" Isnapshot 
if [ [ -z "(snapshot" ]; then 

tffssnap} -i -o mountpoint,backing-store, \ 
backing-store-len tfilesystem » $ 1og1 
${fssnap] -d $fi1esystem 
fi 

done < $fi1esystems 
fi 
fi 

After making the snapshots, return the database to service either by 
restarting it for cold, or ending backup mode for hot. At this point, the 
backup is done as far as the database is concerned. This process takes 
less than two minutes on the systems utilizing this technique. The 
remainder of the task is to start up child processes that mount the 
snapshots, back up the files, and delete the snapshots when done. By 
spawning child processes to perform the actual backup, we multi- 
stream the task and reduce the wall clock time considerably. 

There are additional benefits. For example, when all the files in 
the file system have been backed up, the snapshot actually becomes 
a liability. Not only does it waste space, but until it's deleted, any 
write activity in the snapped filesystem will cause the snapshot to 
grow'. When the child process finishes and deletes the snapshot, the 


used space is returned and becomes available for other filesystem 
snapshots to grow' into. By starting multiple child processes and 
simultaneously backing up multiple filesystems, the lifespan of 
each snapshot is shorter, the eventual size (when the backup is com¬ 
plete) is smaller, and thus the total space requirement is reduced. 

Benefits for Hot Backups 

I occasionally hear complaints about hot backups. Most often, 
there is the issue of maintaining the archive logs, particularly keep¬ 
ing them together as part of a “backup set". There is also concern 
about what the Oracle Backup & Recovery Handbook (by Rama 
Velpuri, 1997, Oracle Press) describes as the “split block phenome¬ 
non”. Split blocks are (logical) blocks where the front half doesn’t 
match the back half. This can occur when a block is written while it 
was being read by the backup task. 

To ensure that the complete block is in place following a restore, 
logs must be re-played during the “recover” operation. This can be a 
complicated process if your backup has taken hours to run, and you 
can wind up w'ith a significant amount of additional data to main¬ 
tain, Herein lies an additional benefit to using UFS snapshots. 
Because the database is only in backup mode during the moments 
the snapshots are being taken, you have minimal archive log data to 
keep, and because it only takes a moment to make the snapshot, the 
only split blocks can be those that were being written during the 
time the snapshots were actually being made. 

The Oracle Backup & Recovery Handbook indicates that the snap¬ 
shots and hot backups tablespaces should be backed up one by one and 
that you should not batch them together. We deviate from that rule 
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Managing the Snapshots 


Taking, and in general, handling the snapshots involves another 
set of tasks, which typically require a privileged user. The scripts 
must create, mount, and delete the snapshots. In some cases at my 
site, a scheduler that is not managed by the sys admin or root runs 
the backups. For this reason, I incorporated the “sudo” tool 
(http://www.sudo.WS) into the scripts. 

If you are running the scripts from root’s cron tab, or manually 
via an admin, you might decide to avoid this. To do so, just modify 
the “sudo” variable to be blank. Yet another approach would be to 
use Sun’s Role Based Access Control. 

We used the following options during the taking of the snap¬ 
shot: backing-store-name, backing-store-limit, and chunk size. 
The script handles this by using the defined “snapfs” filesystem 
and generating a descriptive name; no limit is specified, and the 
chunk size is specified to that of an Oracle logical block. The back¬ 
ing store file name looks similar to other temporary file names 
containing date, time, file system, and other information, and it can 
be useful during the initial setup of the tools. 


I recommend (and the scripts specify) no limit when using a “pri¬ 
vate” snapshot filesystem because any limit can cause an artificial 
problem under these circumstances. If you have to use a “public” 
filesystem, you should read about snapshots before using them, and 
you will probably want to consider limits. As for the chunk size — the 
man pages define this as the granularity of which filesystem blocks are 
copied. The default is four times the filesystem block size. 

If you take the default, it probably won’t be the same as an 
Oracle logical block, and you may wind up wasting space during a 
copy-on-write. Thus, the script queries the database for its logical 
block size and then sets the chunk size equal to that. 

When using UFS snapshots, you must be aware of how much 
buffer cache is allocated for use. By default, the limit is 2% of the 
available physical memory, and you modify this by specifying the 
“bufhmw” parameter in the /etc/system file. This parameter is in 
kilobytes and requires a reboot to change; if you are unfamiliar 
with the details, I recommend using the default. 


because to perform the backup properly via snapshot, all the files in a 
filesystem must be in backup mode before taking the snapshot. The 
“backup” is then logically being made (from the database’s point of 
view) when the snapshot occurs, not while it is being copied to tape. 
This is important to remember, because the backup is not truly com¬ 
plete until all the data has been copied to tape, and there are pluses and 
minuses to this effect. If your snapshot fails (filling the backing store 
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file system, system crash, snapshots don’t persist across a reboot, etc.), 
you must start over. But if the backup server fails during the backup, 
you can pick up where you left off. manually (or scripting) backing up 
the remaining files when you get into the office — provided that you 
built your snapshot file system with sufficient free space. 

Profiling and Space Needs 

The considerations for using this technique (and these two scripts) 
include how you “profile" your Oracle environment, the space needed 
to hold the snapshots, and taking the snapshots. Our site uses the “ora¬ 
cle” userid and a “newdb” prep script to set up the environment. For the 
purposes of these scripts. I’ve included the following example of what 
is needed, namely the PATH information and database SID. If you have 
mother id, or you profile things differently, you will need to either add 
this to your environment in the oracle userid profile, or as a script 
sourcing it in, or make some slight changes to the “parent” script. The 
backup.parent script (Listing 1) contains a $(prep_script) variable 
(located right below the Oracle userid variable), which is used to call 
the setup routine. If your situation doesn’t need this, blank it out. An 
example of what is needed in the setup routine is: 

export 0RACLE_SID=I1 

export ORACLE_HQME=/opt/oracle8/product/8.1.7 
export PATH=${PATH]:${ORACLE_H0ME}/bin 

The space required to hold the snapshots is based on the rate of change 
that occurs in the filesystems being snapped (including non-database 
files). In my experience, allocating a separate filesystem of approxi¬ 
mately 5 to 10 percent of the size of the combined files should be suffi¬ 
cient to start. The only way to determine what is right for your site is to 
try it. Fortunately, you don’t actually need to perform a backup — you 
can manually make a snapshot, estimate the time it would take to back 
it up, then wait and query the snapshot for its “backing store” size. 

If you have a “logical volume manager” and the necessary free 
space, 1 recommend using that to build a permanent backing store 
filesystem and keep nothing there but snapshots. The scripts use a 
/fssnap file system; be sure that you change that to suit your instal¬ 
lation. Because they are transient objects — only good for the life of 
the backup — the backing store filesystem itself can be transient if 
it contains only snapshots. You can create and delete it at will, and it 
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probably does not need to be RAID protected. (There is the slight 
chance your backup could fail, but you must determine whether that 
justifies the additional disk I/O overhead or space requirements.) 
And. if the space requirements change, you can easily destroy and 
recreate it without needing to back up the files contained within. 

The most important consideration when allocating space for a 
snapshot tiles system is scheduling of the backup job. If the job is run 
off-shift, outside of a window used for updating the database, the space 
requirements will probably be minimal. If the database in question has 
a significant amount of write activity during normal operations, you 
will need to test the process to determine the space requirements. In 
my experience, the more write operations, the faster the snapshot 
grows, and the slower the backup runs, thus leading to something that 
looks like a geometric progression of space requirements. 

Specifying the chunk size properly can help minimize the space 
needs. Spreading the database over several file systems helps even 
more, because as each tile system is backed up and the associated snap¬ 
shot is deleted, that space becomes available for the remaining ones to 
grow into. The scripts will calculate and set chunk size and manage the 
snapshots, but the DBA and admin are the ones responsible for select¬ 
ing the schedule. See the sidebar, “Managing the Snapshots”. 

Backing store sizes (Figures 1 and 2) indicate the sizes and distri¬ 
bution oi the 500 snapshots taken for the backup jobs run durins a 
six-month period. Less than 20% of them exceeded 10MB in size, 
and roughly 5% exceeded 1GB. The few occasions where they did 
grow' into the gigabyte range were during periods where the backup 
either ran into early morning (prior to 8:30AM) user activity, or w^as 
run during the day. Given there are 7 filesystems holding the data¬ 
base files, getting 95% of the backups done with less then 7GB of 
backing store space makes it really cost effective. 

TSM Considerations 

As I mentioned, w^e used IBM's TSM product for doing back¬ 
ups. When doing backups, we used the archive command versus 
the backup command to more easily manage the cycles and reten¬ 
tion. (Archives are a point-in-time copy for as long as the expiration 
date specified, unless deliberately deleted.) 

There was one parameter that we modified from the defaults — 
“resourceutilization 10” was added to the client dsm.sys file. This para¬ 
meter allows more streams (sockets) between the client and server; you 
may or may not need to use this. The reference describes that specifying 
”10’ w'ould allow up to eight simultaneous client connections to the 
server. The script defines the number of concurrent child processes equal 
to the number of processors in the host machine, but also provides a 
command-line option to override that default. If your environment 
pusliLS that envelope or you just want to ’‘throttle it . you can either pass 
it the number of child processes to run or modify the script. 

I have other backup products at my disposal and have deliberately 
coded the scripts so that the preferred backup product can be used. 
There aie thiee areas where this comes into play: the first is at startup 
checking to make sure the server is functioning; the second (immedi¬ 
ately following) is verifying the retention period for the “archive” is 
valid (these two could be modified or omitted, if desired): and the third 
( in the child script) is the actual backup itself. 

You can substitute the appropriate syntax for using Legato 
Net worker’s “save". Veritas Netbackup’s “bpbackup”, or any other 
network backup product you want in the “child” script. The scripts 
generate a unique name for each backup run. and use it in the TSM 
archive command’s “description” field. The description name is 
based on the database SID. type of backup (hot or cold), the date, and 
time, and process id of the parent task. This description name is also 
incorporated directly into the file name of the log. This makes for 


simpler management of the backups when performing a query or 
restore. For other products, this would translate to Networker’s “save 
set id” field and Netbackup’s "keyword” field. 

Restore Operations 

To actually restore a tablespace (or the entire database), remem¬ 
ber that the mount point from which it was backed up is where the 
snapshot was mounted, not where the tablespaces are mounted. 
When doing the restore, you must specify the two locations; for 
TSM, it is "file backed up” and the “destination”. Networker uses a 
relocate option, and Netbackup has a "rename file” where the 
backup file and "restore name” are separated by a single space. 
With TSM restoring a single file is a relativity straightforward task. 
When you restore the whole database, it becomes a bit more com¬ 
plex. However, the syntax is still simple, you just need to specify 
each filesystem on a separate command line: 

dsme retrieve -replace-yes -desc=dbname.type.date.time.pid \ 
"Vsnapshot/filesysl/*" "/filesysl/” 
dsme retrieve -replace-yes -desc-dbnane.type.date.time.pid \ 
’Vsnapshot/fi1esys2/*" "/filesys2/" 

dsne retrieve -replace-yes -desc^dbname.type.date.time.pid \ 
"/snapshot/filesysN/*" "/filesysN/" 


While I have done this for small databases, I have not yet had to restore 
an entire large database, thus can't accurately estimate the time to do 
so. Things to take into consideration are that TSM (and probably oth- 
eis) tend to store tiles in an online storage pool during the backup 
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process, and then migrate them to tape when resources permit. Thus, 
while you may be able to start multiple restores, many of the restore 
requests will likely queue up against the same physical tape media. If 
you do implement this process, insure that you set a reasonable level of 
expectation on when such a restore might complete. 

Conclusion 

While no process is foolproof, these scripts can simplify the task 
and allow' it to be automated to am during the best times for perform¬ 
ing the backup. These scripts will determine which files are to be 
backed up. execute the appropriate commands to ready the database 
(be it shutdown for cold, or backup mode for hot), snapshot the 
filesystems, bring the databases back online, and run the backup tasks 
in parallel to minimize the w'all clock time. (The latter presumes the 
target system is doing some data compression, and has multiple CPUs 
with which to do so.) Diagnostic information is written by both 
scripts into the /tmp filesystem, collected at the end of the run and 
moved to a log directory (the default being adjacent to the location in 
which the scripts are placed). During the run. interesting messages are 


written to the system log using the logger command, having first 
added a “local 1.notice" facility entry to the /etc/syslog.conf file. 

When doing some tuning. I've found the following information 
helpful: 

* netstat -k has a "biostats" section that reports on hits and 
lookups in the cache. 

* kstat -p uni x:: var: v_bufhwm seems to provide the upper limit 
in K. 

• mdb -k with a bfreel ist$<buf command returns the number of 
buffers not yet allocated in the bufsize field. 

• sar -b will produce statistics on the buffer cache. 

Rick Moak ley is a Unix Systems Administrator with Epsilon Data Mgmt. 
Over the past 25 years, he has served in Operations, Systems Programming, 
Software Engineering, and Process Improvement teams with an array of 
roles from managing and maintaining mainframes and Unix systems, to 
developing applications in System 390 assembler, C, Lotus Notes, SA S, and 
more. During his free rime, he likes "to putter around" with home projects, 
and spend time with his fiance. 


Listing 1 backup.parent script 


■m 





(M/bin/ksh 


it Version 1.1,0 

# 

# 01/12/03 ROM Tivoli Storage Manager Backup for an Oracle database whose 

if files reside in a UFS filesystem 

# 

if This "job" is made of two seperate scripts. The "parent" 

if that performs management and control operations (ie: get 

if oracle file lists, db startup/shutdown, etc.) and the 

if "child" that gets tasked to perform the ”dsmc archive" 

if tasks themselves. 
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if 
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This script will backup an Oracle database using a TSM 
server. The filenames are retreived from the Oracle 
database itself at runtime. A set of child processes 
perform the actual "dsmc archive" tasks to allow mulrf- 
streaming operations. 

example syntax to run: 

./backup.parent dataoase.sid HOT 2 ARC2WEEKS 

Motes; This script utilizes Solaris filesystem snapshots when 
backing up the files, thus minimizing the time that the 
database is down (for COLD) or in backup mode (for HOT) 

The database is taken down (altered, etc.) and when 
it Is ready to be backed up. a "snapshot" of the filesystem 
is taken (see fssnap command.) Once the snapshot is done 
the database can be restarted (etc.) A seperate filesystem 
is used to store the snapshot "backing store" files, 
and the snapshot (ie; /dev/fssnap/3) is mounted (in our case 
in the /tmp filesystem.) When the "dsmc archive" task runs 
it is pointed at the snapshot. Provided that the amount of 
changes to the snapshot'ed filesystems do not exceed the 
fssnap filesystem capacity the archive task should run 
normally. In the event that the snapshot filesystem is 
overloaded the snapshot being written to fails, and is deleted. 
Please refer to the fssnap and fssnap_ufs manpages. 

The default for a backup is COLD • if you do not wart the 
database shutdown make sure you specify HOT. 


The default snapshot filesystem will need to Pe modified 
see the "snapshot_fs" environment variable below. Vou will 
get a warning message if it is not mounted. 


Since we are using filesystem snapshots and the /tmp filesystem 
becomes the mount pcint "holder", all the files (with the 
exception of the "archive logs" in the case of a HOT backup) 
will have been backed up from the /tmp filesystem from the 
viewpoint of TSM. Therefore you will often need to prefix 
pathnames with /tmp to get the proper pathname. 

A presumption of the "oracle" user environment exists. 

The user is "profiled/setup" by locally used routines, that 
may not exist in your environment. The QRACLE_SID, ORACLE_HOME, 
and PATH (appending $ORACLE_H0ME/bTn} environment variables 
must be prooerly defined for the SQL*Plus commands to work 
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it 

it 

it 

it 
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it 


correctly, if they are not defined in the oracle userid profile 
you can use the $lprep_scriptj variable to execute the commands 
or "source them in". See your DBA if you have questions. 

The use of "here is" documents (ie; some_cmd << EOF) is 
used to instream the connections to the database to perform 
queries etc. During the processing of this script several 
times oracle commands and info are handled by using the 
"su - oracle" command, feeding it the commands (typically 
"newdb", and "sqlplus” w/sql cmds) this way, often nesting 
some of them. Normally one can place a hyphen after the 
redirection characters to strip out leading tabs and allow 
indentation for readability, I have deliberately not done so 
for the benifit of those that would end up copying the info 
and converting the tabs to blanks. 

the scripts perform the following operations; 

1) get the names of the files needing backup from tne 
owning database. 

2) separate the files by filesystem into discrete "control" 
files (to be passed to the "archive" child processes.) 

3) create a "queue" of tasks for background (child) processes. 

4) ready the database for backups. 

5) snapshot the filesystems that hole backup targets, 

6) ready the database for users. 

7) run archive task for each filesystem. 

a) run up to tM tasks so as to utilize all epus. 

b) cleanup/delete snapshots 

8) capture logs 

NoteZ: 

1) Snapshots (and their backing store files) are not deleted if 
archive task fails - and must be manually deleted before the 
snapshot file system becomes full, (see Tip #1 below) 

2) A "resDurceutilization nn" staement may be required in the 
dsm.sys (TSM system options) file, and may also require a 
change to the TSM server (for the target node) for more 
"direct tape sessions" if you go direct to tape for large files 
if reeded use "MaxNumMp" setting in the server config, 


Archive maintenenee: 

To delete old copies use the "dsmc delete arch" command 
passing it each mountpoint that was backed up and the description 
field corresponding to the level at which you want the deletion 
to occur (ie: database.day.time.process ■ note the time and pid 
should be unique.) 
ie; 

dsmc delete archive /tmp/sharkOl/ -desc="testdb.d030120.*" 

meaning delete all files that were archived from the /tmp/sharkOl 
mount point on 01/20/03. Note that you will have to do this for 
each mount point 


it Tips/Notes/Warnings; 
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Listing 1 continued 







it 

# 1) If the backup of a file fails during the running of the script 

# you can attempt another archive of the file provided the snapshot 

# still exists. (The child processes attempt to check the return 

# codes from dsmc and not delete the snapshot if the archive 

it fails.) Use the "dsmc archive" command manually to finish the 
it failed backup, (see below) 

it 

it 1) find log of backup 

it 2) grep for "Unsuccessful" 

it 3) make sure the snapshot is still there and mounted 

it 4) do a manual archive(s) 

it 5) remove the backing store file 

it 6) unmount the snapshot 

it 1) delete the snapshot 

it 

it find . -name "backup.parent.*.logl" -mtime -1 -print 

it ./backup, parent, criisdb. COLD, d030127.tl03713.p8242.logl 

it 

it 

it 

it . /ba ckup. pa rent, cri is db. COLD. d0301 £8. t040000.pl 7 556. log 1 
it 

it grep -1 unsuccessful 

./backup.pa rent.criisdb.COLD.d03G128,t040QQ0.pl7556.logl 
it Normal File--) 20,971,536,384 /tmp/shark04/oradata/data_105.dbf \ 
** Unsuccessful ** 
it 

it fssnap -i -o blockdevname awk -F: ’ {print "df -k",(2]' | sh 
it Filesystem kbytes used avail capacity Mounted on 

# /dev/fssnap/2 64225208 52550912 11674296 82% /tmp/shark04 

it 

tt dsmc archive /tmp/shark04/oradata/data_105.dbf -filesonly \ 

tt -desc-"cri isdb.C0LD.dQ30128.t040000, pl7556" 

tt 

it fssnap -i -o backing-store j awk -F: ’{print "rm".(2]' 
it rm /fssnap/backup. parent, d030128.t040000.pl7556._sha rk04. bs 
tt 

it umount /dev/fssnap/2 

tt 

it fssnap -d /dev/fssnap/2 

it Deleted snapshot 2. 

tt 

it 2) To get a list of files that have been archived (or backed up) 
it may require them to be listed according to the mount point on 

it which they were backed up. This may be difficult for the 

it snapshot'ed filesystems as they are transitory. Try doing a 

it "dsmc query filespace" and look for objects that were backed 

it up in the /tmp/* filesystem names: 
it 

it dsmc query filespace | grep "/tmp/" 

it 

tt 37 00/00/00 00:00:00 UFS /tmp/sharkOl 

tt 

tt 

it dsmc query filespace j grep "/tmp/" | awk '{print "dsmc query archive V" 
it (5 ’7*\" -desc“\"cmsdb.CQLD,dQ30128.*\""}' 

it dsmc query archive "/tmp/sharkOl/*" -desc-"criisdb.C0LD.dO30128.*" 

it dsmc query archive ’7tmp/shark02/*" -desc="criisdb.C0LD.d030128.*" 

tt 

it 3} To restore a file, will require that you do a "dsmc RETrieve" 
it of the files as they were backed up (ie: /tmp/sharkOl/.,,) 
it using the rename option to put them in the proper location. 
it ie: 


it maestro pill-NOPASSWD: /usr/bin/rm /tmp/backup*, /usr/hin/mv /tmp/backup* 
it maestro pill-NOPASSWD: /usr/bin/rm /fssnap/backup* 
it maestro pill-NOPASSWD: /usr/bin/dsmc ouery, /usr/bin/dsmc archive 
it 

it . options passed 

DB-1I 

hot_or_cold-$2 
children=$3 

retention_period-(4 

it . verify options and/or set defaults 

if [ -z "tDB" ]; then 

echo "No database specified" 

echo "Syntax is:” 

echo 

echo to "database_sid [ COLD | HOT ] [ num_streams ] [ retention_period ]" 
echo 
exit 4 
fi 

it type of backup COLD or HOT 
hot_or_cold-$[hot_or_cold:-"COLD") 

if [[ $hot_or_cold !- "COLD" M (Hot_or_cold 1- "HOT" ]]; then 
echo "Error \”$hot_or_cold\" not valid” 
echo "specify COLD or HOT" 
exit 4 

fi 

it default number of tasks, when not specified, psrinfo and word counting the 
it number of lines returned is used to determine the number of cpus. if this 
it does not work specify the default manually. 
tt 

:hildren-((children:-((psrinfo | wc -1)) 

it MGMTCLAS option for retention information, you can determine available 
it classes by doing a "dsmc query mgmtclas" command 
retentf on_period—i[retention_period:-"ARC5WEEKS") 

export DSM_DIR-/opt/tivoli/tsm/client/ba/bin 
export DSM_CONFIG=${DSM_DIR]/dsm.opt 

export PATH=/usr/sbin:$(PATH}:${DSM_DIR) 

it . functions 

function keep_going { 

it request permission from operator to continue 
echo "reply \"yes\" to continue" 
read RESP 

if [ "$[RESP1" !- "yes" ]; then exit: fi 

} 

it . variables (filenames, etc.) 

it the child script that performs the actual "dsmc archive" task 
\ it (this script should reside in the same directory as this parent script) 
chi 1d-iCdirname (0)/backup.child 

tt a set of unique names for temporary files is generated by using the /tmp 
it filesystem, name of the script, date 4 time, and finally'process id (pid) 

timestamp-di(date +SEyMd) . ti f date +SHSMSS) 
tmpname-/tirp/$(basenaire $0).$DB.$hot_or_cold.$timestamp.p$$ 


it 

it dsmc retrieve -desc-"criisdb,CQLD.dQ30128.*" \ 

it /tmp/shark04/oradata/temp02.dbf /shark04/oradata/temp02.dbf 

it 

it Insure when you do this that there is ONLY ONE file that will be 
it "retrieved" by doing a query first,., 


it --- 

it Maintenance: 
it . 


it 

it 02/11/03 RCM 

it 

it 


Added dsmc "mgmtclas" option for specifing backup retention 
You can determine which classes are available for use by 
doing a "dsmc query mgmtclas" command, 


it 04/14/03 RCM Utilizing "sudo" to run fssnap, mount, and umount commands. 

it Normally these require root authority, and setting the uid 

it bit on in the permissions is a) not good, and b) seems to 

it create a problem by which to becomes the shell name (ie: ksh) 

it While not a crisis, it is not what was desired, therefore 

it we will define (fssnap, (mount, (umount variables that have 

it the appropriate values. The /etc/sudoers file will need the 

it following (uncommented) lines: 

it 


it maestro pill-NOPASSWD: /usr/sbin/fssnap, /usr/sbin/mount, /usr/sbTn/umount 
it maestro pill-NOPASSWD: /usr/bin/su - oracle 


child_prefix=/tmp/$(basenanie tchild).(D8.(hot_or_cold.(timestamp 
it (the child's pid will be appended in the (child script) 

tt description is the "description" option (field) that is passed to the 
it tsm server when the file is archived. This field can be used as a 
it "keyword" field when listing and/or deleting old backups, 
descript1on=(D8.(hot_or_cold,(timestamp,p$( 

it files used by the script 
tmpfilel-ttmpname.tmpfi lei 
tmpfile2-ttmpname.tmpfile2 
tmpfile3-(tmpname.tmpfile3 
tmpfi1e4-$tmpname.tmpfi1e4 
logl=(tmpname.logl 
log2=$tmpname.log2 
fllesystems-Umpname.filesystens 
begin_bkup-$tmpname.begin_bkup 
end_bkup-(tmpname.end_bkup 

it log directory, if it doesn't exist create it 
logdir=((dirname (0)/../logs 
if [ ! -d ({logdir} ]; then 
mkdir ({logdir} 
fi 

It 04/14/03 RCM Command variables, used for using "sudo" with the following 
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Listing 1 continued 


commands: (sudo,) fssnap. mount, umount, su - export them so 
that child processes, will see them properly (su need not) 
export sudo-"/usr/lcca1/b1n/$udc " 
if [ ! -x J {sudol ]; then 
export sudo=”" 
fi 

export fssnap—’“t(sudo}/usr/sbin/fssnap" 

export mount="${sudo)/usr/sbin/mount" 

export umount="J(sudo)/usr/sbin/umount" 

su-"$lsudo}/usr/bin/su" 

rm="t(sudo}/usr/bin/rm" 

mv=”J (.sudo}/usr/bi n/mv" 

export dsmc-"J{sudo}/usr/bin/dsmc" 

it Oracle admin id and prep script (profile the environment) 

# Note that you will have to decide which approach for profiling the oracle 
userid works best for your environment. You may decide to imbed the 
commands for exporting the database ORACLE^SID, ORACLE_HOME, and PATH 
environment variables in a .profile, .khrc, etc. file or "source" 

them 

in via the prep script if you have multiple databases. (Our "newdb" 
profiling script/function is much more complex than what is needed for 
doing just a backup. 
oracle-"oracle" 
prep_script=”newdb JDB" 



it Make sure we can talk to the server. 

J[dsmc] query session >> Jlogl 
if [ $? -ne 0 ]: then 

logger -ip local 1.notice JO": Error, Can not establish session with \ 
server properly" 

logger -ip locall.notice JO": exiting" 


echo "Error. Can not establish session with server properly" 
exit 


fi 


it Check to make sure we can use the management class specified 

Hdsmc] query mgmtclas J grep tfretention_period} 

rc=J? 

if [ $rc -ne 0 ]; then 

logger -ip locall.notice $0": Error. Storage MGMTCLAS 
t[retention_period) invalid" 

logger -ip locall.notice JO”: exiting." 

echo "Error, Storage MGMTCLAS ttretention_period} invalid" 
exit 
fi 

if filesystem where the filesystems of the target files to be backed up, can 

# be snapshot 1 ed Me: the "backing store” filesystem) 
snapshot_fs-/fssnap 

snapshot„fs_line=t(expr J(LINENO) - 1) 

# Check to make sure that our "snapshot„fs" is mounted. 
host_fs=J(df ${snapshot_fs} | cut -fl -dV | tr -d ’ ') 
rc*J? 

if [[ $rc -ne 0 |' $(.host_fs) !- J(snapshot_fs) ]]; then 
echo 

echo "Error, the snapshot filesys (tisnapshot_fs)) is not mounted." 
echo "" 

echo " insure that this is the correct file system, and it is mounted" 
echo " or change the "snapshotjfs" envinonment variable in the JO" 

echo " script at line number J(snapshot_fs_line}" 

echo "" 

logger -ip locall.notice $0": The snapshot filesys: J(snapshot_fs} not mounted." 
logger -ip locall.notice JO": backup canceled," 
exi t 
fi 

if since the backing stone files are stored in a different filesystem, 

# they need a different naming convention. 
bstmpname-Jsnapshot_fs/J(basename JOMtimestamp.pJS 

if we export these so that awk process (below) can see it. 
export tmpfi1e3 tmpfileA 

export CTLFlLE_BKUP=Jtmpname.ctlfile_bkup 
export CTLFILE_BKUP_NAME“\'JCTLFILE_B<UP\' 

# . main routines 

logger -ip local 1.notice JO": Oracle database (SDB) backup started" 

if "View" names that we will use to gather info from/about the database 
j) We will need the datafiles for starters, and depending on the type 
if of backup (cold/hot) we will need the controlfiles, logfiles, and 
if archive logs. 

if Note: on processing/passing view names... 

if Insure that you escape the dollar sign that oracle needs for the "sysadmin" 


if views, by imeadiately prefixing it with a back slash, or the shell will 
it think that it is a variable to process. 

CONTROLFILE“'v\tcontrolfile' 

LOGFlLE-’v\$logfile" 

DATAFILE-’vUdatafile' 

TEMPFlLE='v\$tempfile' 

ARCH I VED_LOG=' vUarchived_log’ 

if To obtain the "critical” database files, we run SQL*Plus and run selects 
if to query the database system views (or in the case of the datafiles, at 
if the request of one of our DBAs the sys.dba_data_files table... - feel free 
if to set it back to the view.) 

if the select statements format the output for later use by using colons for 
if field delimiters, and other text for describing the record types. 
it the format is: 
it 

it column 

it 1 filetype identifier (ie: CONTROLFILE, TABLESPACE, etc.) 

it 2 "tablespace name” if any (only used for tablespaces) 

# 3 os filesystem pathname of the data (our backup target files) 

J{sul - SloracleJ « EOF » Jlogl 2H1 
${prep_script] 

sqlplus -SILENT "I as sysdba” « EON » JlogZ 

whenever sqlerror exit 1 

set pagesize 0 

set heading off 

set timing off 

spool Jtmpfilel 

select 'CONTROLFILE::' 11 name from JCQNTROLFILE; 

select 'TABLESPACE;'[|tablespace_name||||file_name from sys.dba_data_files; 

select 1 LOGFILE::* j jmember from JLOGFILE; 

select "TEMPFILE::'j|NAME from JTEMPFILE; 

spool off 

set echo off 

spool Jtmpfi1e2 

SHOW parameter db_block_size 

spool off 

exit 0 

EON 

EOF 

if t J? -ne 0 ]; then 

echo “Error getting tablespace info (from "JOB”)" 
logger -ip locall.notice JO": Error getting tablespace info (from "JOB”)" 
exit 
fi 

cat Jlog2 » Jlogl 
${rra} Jlog2 

if read in the datafile (logical) block size 
read name datatype blksize < Jtmpfi1e2 
echo "db_block_size=" Jblksize 
J{rm] Jtmpfile2 

it go through the path lists, and generate input for the child 

if dsmc backup jobs, the pathname is the last field in our temp file - awk's 

it JNF "variable" ("NF" is the "Number of Fields" on the input record, and 

it thus if NF—3 then JNF-J3) 

it 

it the arguments the child tasks will receive are: filesystem name, pathname 
if of a file that contains the pathnames within that filesystem of the backup 
it targets, and the fssnap "snapshot" device name (ie: /dev/fssnap/3) 
it assigned to the filesystem. 

it we seperate the list of pathnames to be backed up into seperate files 
it by filesystem name, then pass those file names to each child process child 
iff started, thus each child process backs up only those files within it's 
if designated filesystem. 

it we determine the filesystem name by doing a ”df" on each file then write 
if the "target" pathname to a file named after the "target filesystem", also 
it writing the name of the filesystem and it's associated "file lists" 
if pathname to a seperate file, we sort/de-dup that file, and use it to 
it schedule the child processes by filesystem. 

it Note: we generate discrete files for each filesystem by "echoing" the 
it the pathname of each target file (those being backed up) to a file whose 
if name incorporates the filesystem of the target file, thus each time the 
it filesystem changes, the pathname of the target file is written to a "new" 
it file. 
it 

it * .>..>. + 

i i \ 

it / \ \ 

it ie: /sharkOl/oradata/dataJOl.dbf + 

it 

it is written to a file named: 
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Listing 1 continued 


ft / \ 

ft /tmp/backup,parent.d03CH14.t093015.pl920._shark01 
ft 

ft thus when the above files are created all the backup targets in "/sharkOl" 
ft filesystem will be in the "*._shark01" file, the targets for "/sharkQ2" 
ft will be in "*._shark02", etc. these files then become the file lists for 
ft the "dsroc archive" child processes. 
ft 

ft the underscores are used to replace the slashes {"/") in the file 

ft names to create "valid" pathnames, ie: /opt/oracle becomes _opt_oracle 
ft this simplifies things considerably. 
ft 

ft since we do this for each pathname to be backed up, we wind up with a 
ft "control" file (one that we will use for queuing up multiple streams) 
ft that contains duplicate entries ie: five files being backed up, existing 
ft in three different filesystems will produce a five line file, with two 
ft lines repeated, (see example below) 
ft 

ft /sharkOl/oradata/init_load„01e.dbf 
ft /shark01/oradata/init_load_02.dbf 
ft /shark02/oradata/indx_s01,dbf 
ft /shark02/oradata/init_load_G1g.dbf 
ft /shark03/oradata/data_l03.dbf 


would generated the file: 

/sharkOl /tmp/backup.parent.d030114.tl05520.pl920._shark01 
/sharkOl /tmp/backup.oarent.d030114.t105520.pl920._shark01 
/shark02 /tmp/backup.parent.d030114,tl0552Q.pl920._$hark02 
/shark02 /tmp/backup.parent.d030114.tlQ5520.pl920._shark02 
/sharkD3 /tmp/backup.parent.d030114.t105520.pl920._shark03 


ft 
ft 

ft sorting/de-duping produces: 
ft 
ft 
ft 
ft 
ft 


/sharkOl /tmp/backup.parent.d030114.tlQ5520.pl92Q._shark01 
/shark02 /tmp/backup.pa rent,d030114.1105520.pl920._shark02 
/shark03 /tmp/backup.parent.d030114.t105520.pl920._shark03 


ft now we can spawn one child "dsmc archive" task for each line in the above 
ft file, thus giving us the ability to rnulti-stream the archive tasks. 

ft we use "grep" to get only the desired data, and pipe it to awk to print 
# the last field of the output from the oracle selects for filenames, (ie: 
ft this is the pathname of the file to be backed up,) thus pathlist becomes 
ft a list of the backup targets. 

ft Note: L0GFILES (redo/archive) have a different structure. These files 
ft must not be backed up using snapshots, and redo logs should not 
ft be backed up during a HOT backup. 

ft 

ft Also TEMP files should rot be backed up either. Prior to Oracle 8 
the tempfiles were needed to bring up the database. Also they can 
be "sparse files" that can cause problems with backup products. 

if [ $hot_cr_cold — "HOT" ]; then 

ft 07/30/03 RCK grep'ing out Only the tablespace files (archive logs and 
ft control files must be processed differently.) Temporary 

ft files are not to be backed up in ary case, but we keep 

ft the names here for documenting the fact they exist. 

pathlist=((grep ^"TABLESPACE:" (tmpfilel | nawk -F: ’[print (NFJ’) 
pathlist-$(grep -v ""TEMPFILE:" (tmpfilel 


el se 
fi 

for path in (pathlist; do 

fi1esys=$(df (path|cut -fl -d'(’|tr -d ' ') 

outfi1e—J(echo (fi1esys|tr / _) 

echo (path >> Hmpname.(outfile 

echo $fT1 esys (tmpname.(outfi1e » (trnpfi 1 e£ 

done 

ft sort and de-dup the temp file 
sort -u -o (filesystems (tmpfi1e2 
(f rm] $tmpfi1e2 

#keep_going 

for COLD backup: 
ft connect to the database ana shut it down 
if [ ihot_or_ccld - "COLD" ]; then 
ifsu] - oracle << EOF >> (logl 2>&1 
${ p r e p_s c rip t J 

sq1 pi us ”/ as sysdba" « EON » $1og2 

whenever sqlerror exit 1 

set timing off 

shutdown immediate 

exit 0 


nawk -F: '{print (NF) ’) 


EON 

EOF 


if [ (1 -ne 0 ]; then 

logger -ip localL.notice $0": Error shutting down database:" (DB 
echo "Error shutting down database:" (DB 


fi 


exit 

else echo "Database shutdown, 0k" 

logger -ip locall.notice (0": Database ("tDB") shutdown. Ok" 
fi 

cat 11og2 >> $1ogl 
$(rm) tlog2 


ft for HOT backup: 

ft put the database in backup mode 

if [ thot_or_cold - "HOT" ]; then 

ft set up two files, one with the commands to place the database in 
ft backup mode, the other to take it out of backup mode 
ft Note that where tablespaces occupy multiple files, the duplicate 
ft tablespace names need to be removed. 

grep "TABLESPACE (tmpfilei j cut *f2 -d: | sort -u -o $ tmpff1e2 

ft generate BEGIN BACKUP mode commands 
nawk -F: \ 

'BEGIN { 

print "whenever sqlerror exit I" 
print "set timing off" 
print "spool", ENVIR0N["tmpfile3”] 
print "show parameter 1og_archive_dest" 
print "archive log list" 
print "spool off" 

) { 

print "alter tablespace",$1."begin backup;" 

} END { 

print "exit 0" 

] 1 (tmpf11e2 > $ begin_bkup 

ft generate END BACKUP mode commands 
nawk -F: \ 

'BEGIN { 

print "whenever sqlerror exit 1” 
print "set timing off” 

1 { 

print "alter tablespace".$1,"end backup;” 

} END ( 

print "spool", ENVIR0N["tmpf1le4"] 
print "archive log list" 
print "spool off" 

print "alter system switch logfile;" 
print "alter database backup controlfile to \ 

" ENVIRON["CTLFILE_BKUP_NAHE"] 
print "exit 0" 

1’ (tmpfT1e2 > $end_bkup 
l(rm) $tmpfile2 


i{su) - ((oracle) << EOF » tlogl 2>&1 
${prep_script} 

sqTpi us “/ as sysdba" < (beginjikup » $log2 
if [ i? -re 0 ]; then 

logger -ip 1 ocaT1.notice $0": Error putting database in backup \ 
mode:" (0B 

echo "Error putting database in backup mode:” (CB 
exit 


EOF 


fi 


else echo "Database placed in backup mode, 0k" 

logger -ip locall.notice (0": Database ("(OB”) placed in backup \ 
mode, 0k" 
fi 

it go through the "spool” file that contains the "archive log list" output 
ft and determine the archive directory and the sequence number of the 
ft "oldest" archive log file. 

arch_dir=((grep -i ""log_archjve_dest " Jtmpfile3 | awk '(print INF)’) 
echo "archive dir is:" (arch_dir 

oldest-((grep -i ""Oldest " (tmpfiTe3 , awk '{print (NF}’) 
echo "oldest sequence number is:" (oldest 
cat $log2 >> $ 1ogl 
(f rm} $1og2 sbegin_bkup 


#keep_going 

ft read the filesystem names and snapshot each one re-writing the (filesystems 
jjj control file with the snapshot device name included to each record. 

ft Note: If doing a HOT backup, the archive logs can not be snapshot'ed and 
ft tfien backed up, we will add them to the list below with /dev/null as 
ft the name of the snapshot so the child process handling it knows. 

ft Use db_block_size for chunksize 
chunks)ze=(fexpr (blksize / 1024)k 
while read filesystem pathnames 
do 

outfi1e-((echo (filesystem|tr / _) 
snapshot-(((ffssnapf -F ufs \ 

_ backing-store-tbstmpnaroe.(outfile.bs.chunksize=(chunksize \ 
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$fi7 esystem) 

rc-t? 

if [ Ire -ne 0 ]; then 

logger -ip 1 oca 11.notice 10”: Error" tre "doing snapshot for:" \ 
t fi1esys tern 

echo "Error" $rc "doing snapshot for:' 1 (filesystem 
# ernor? don't exit here (database status?) see below 
break 

else echo $filesystem "snapshot taken, Ok" 

logger -ip local 1.notice JO":" $fi1esystem "snapshot taken. Ok” 
fi 

echo (filesystem Spathnames (snapshot >> $tmpfi1e2 
done < S fi1esys terns 
({rm} (filesystems 
$(mvl Stmpfi1e2 Sfilesystems 

if [ ire -ne 0 ]: then 

# some snapshot problem must occurred, delete the ones that we 
it have created so far and bring the database back up before exit 
if while read filesystem pathnames snapshot 
echo "Snapshot ernor seems to have occured, deleting snapshots” 
df -k » (logl 

if [ -f t fi1esys terns ]; then 

while read filesystem pathnames snapshot 
do 

logger -ip local 1.notice SO"; Removing snapshot" (snapshot 
echo "Removing snapshot" (snapshot 
if [ 1 -z "(snapshot" ]; then 

([fssnap] -i -o mountpoint,backing-store,backing-store-len \ 
(filesystem >> (logl 

if 07/24/03 snapshot device name not working so well under Sol8 

# (snapshot >> (logl 

# $ {fssnap"} -d (snapshot 
([fssnap} -d (filesystem 
fi 

done < (fi1esystems 
fi 
fi 

#keep_going 
it for COLD backup: 

# connect to the database and start it back up 
if [ $hot_or_cold - "COLD" ]; then 

$[su) ■ ({oracle) « EOF » (logl 2X1 
({prep.script} 

sqlplus "/ as sysdba" << EON >> (log2 
whenever sqlerror exit 1 
set timing off 
startup 
exit 0 
EON 
EOF 

if [ (? -ne 0 ]: then 

logger -ip local 1.notice (0”: Error restarting database:" (DB 

echo "Error restarting database:" (DB 

exit 

else echo "Database restarted - users may go back to work" 
logger -ip 1 oca 11.notice SO": Database ("$DB") restarted, Ok" 
fi 

cat (1og2 >> (logl 
([rm] (log2 
fi 

it for HOT backup: 

# take the database out of backup/archivelog mode 
if [ (hot_or_cold - "HOT" ]; then 

({SU} - ({oracle) << EOF >> (logl 2X1 
$(prep_script] 

sqlplus "/ as sysdba" < (end_bkup >> (log2 
EOF 

if [ (? -ne 0 ]: then 

logger -ip local 1.notice $0": Error taking database out of backup \ 
mode;" (DB 

echo "Error taking database out of backup mode:" (DB 
exit 

else echo "Database taken out of backup mode" 

logger -ip local 1.notice (0": Database ("(DB") taken out of backup \ 
mode. Ok" 

# go through the "spool" file that contains the "archive log list 

# output and get the "current" archive log file. 

cat $log2 >> (logl 
({rm} (log2 (end_bkup 

# aurguably we might want to wait for a moment or two in this step to 
if allow time for the the archive log switch command to take effect... 
if I've not seen any problems but if so add a sleep below * like... 

If sleep 10 


if To backup the archive logs according to the Oracle E&R Handbook 
if you need to get all the archive logs that were In use between the 
if the "backup start" and "backup end" commands. We got the "oldest" 
if earlier when we put the database in backup mode by spooling the 
if "console log" and then doing an archive log list and grep'ing for 
if "oldest". Then (just above) after snapshots were taken, we did 
if another "archive log list" (before switching the logs, but after 
If taking the database out of backup mode), and got the "current" log. 
if (See example of "list" below.) 


if SQL> archive log list 
if Database log mode 
if Automatic archival 
if Archive destination 
if Oldest online log sequence 
if Next log sequence to archive 
if Current log sequence 

if Is -It /oraarch/testdb/ | head 
if total 566912 


Archive Mode 
Enabled 

/oraarch/testdb/ 

267 

269 

269 


if -rw-rw— 

1 oracle 

dba 

10485248 May 

# -rw-rw— 

1 oracle 

dba 

10495248 May 

if -rw-rw— 

1 oracle 

dba 

104B5248 May 

if -rw-rw— 

1 oracle 

dba 

10485248 May 


if ... 

if get "Current log sequence" 

current-((grep -i ‘"Current " (tmpfile4 | awk '{print (NF]') 
echo "current achive log is:" (current 

if We have the "oldest" sequence number above, now having obtained the 
if current sequence number and having done the log switch we can query 
if the database for all the logs between the two sequence numbers, These 
if will be the ones that need to be backed up with the datafiles for the 
if "recover" command to work properly. 

({sul - ([oracle] « EOF >> (logl 2X1 
([prep_scriptl 

sqlplus -SILENT "i as sysdba" << EON >> (log2 
whenever sqlerror exit 1 

set linesize 100 trimspool on feedback off echo off term off heading \ 
off pagesize 0 
spool (tmpname.archivejogs 

select name from (ARCHIVED_L0G where sequence# > _ ({oldest} and \ 
sequence# <- ({current}; 
spool off 
exit 0 


EON 

EOF 


fi1esys—$Cdf ({dirname (arch_dir)|cut -fl -d'(') 

echo (filesys (tmpname.archivejogs "/dev/null" >> (filesystems 

if now build the backup cards for the control file we told oracle to "backup" 
pathlist=(CTLFILE_BKUP 

f i 1esys - $ Cdf ((dirname (CTLFILE_BKUP)|cut -fl -d'(') 
for path in (pathlist: do 

echo (path » (tmpname,control_file 
done 

echo (filesys $tmpname,control_file "/dev/null" >> (filesystems 


fi 


if [ (rc -ne 0 ]; then 

# one of the snapshots attempt above failed, in theory we will have 
If deleted the others, and resumed user operations with the database 

# and we can now exit. 

echo "Snapshot error must have occured, exiting" 
logger -ip locall.notice (0": job terminating." 
exit 
fi 

#keep_going 

if read the file back in and run a background job for 
# each filesystem. 

while read filesystem pathnames snapshot 
do 

if if there are max background jobs running, sleep 
if until one ends - before starting any more 
while [ ((jobs -p | wc -1) -ge (children ] 
do 

sleep 5 

done 

# start a background task and tell it which 

if files to process, then add it to the pid_history list 
(child (filesystem (pathnames (chi 1d_prefix (snapshot \ 
(description (retention_period & 
pid history-(pid history" "(! 
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Listing 1 continued 


4 wait a few moments for it to start up, then do art iostat to get 
4 some numbers regarding the i/o throughput (feel free to change or 
# remove the echo and iostat commands below, 
sleep 5 

echo "----" >> Ilogl 

iostat -xtnTd 3 3 » Ilogl 
done < Ifilesystems 
fflirmj Ifilesystems 

# for now save the filesystems list... (We might need it for other stuff) 
Ilmvj Ifilesystems $ 1ogdfr/$(basename Ifilesystems) 

#keep_going 

4 wait for all jobs to end, and then collect all logs 
wait 

for pid in $pid_history 
do 


cat Ichild_prefix.Ipid.log >> Ilogl 
Hrm] Jchildjjrefix.tpid.log 

done 

echo “files archived via adsm:" 
grep -1 normal Ilogl 

echo "\nDescriptior field used:" tdescription 

echo "\n to list files use the mourtpoint name (ie: /trap/sharkOl/)" 
echo "and the -desc=... option to list them, ie:" 
echo "\n" 

echo "dsmc query archive V7tmp/shark01/*V -desc-\""Idescription"\"\n 
4 finally move the logfile to a permanant location, in this case the 
4 /usr/local/logs directory for example. 

Ilmvl Ilogl Hogdir/. 

logger -ip locall.notice $0": Oracle database ("IDB") backup complete 1 
echo "Parent ending" 
rm Itmpname.* 
exit 


Listing 2 backup, child 


#!/bin/ksh 



file system—$ 1 

pathnames-12 

log-I3.SI.log 

snapshot-14 

description-15 

retention_period-I6 


if [ ! -d Itrapmountpoint/Ifs ]; then 

names-Stmpmountpoint/tfs" "tnames 
fi 

fs-Kdirname Sfs) 

done 

if [ Ktfnames} -gt 0 ]; then 
mkdir Inames 
fi 


export PATH-/usr/sbfn:$iPATHf 

4 Command' variables, used for using "sudo" with the following 
4 commands: (sudo,) fssnap, mount, umount, su - export them so 
4 that child processes, will see them properly (su need not) 
export sudo-"/usr/1ocal/bin/sudo " 
if [ ! -x JIsudo} ]; then 
export sudo="" 
fi 

export fssnap-"!{sudo1/usr/sbin/fssnap " 

export mount-"!{sudo]/usr/sbin/mount " 

export amount-"!{sudoj/usr/sbin/umount " 

su="I[sudoj/usr/bin/su" 

rm-'Ilisudol/iisr/bin/rm" 

mv”"l[sudo}/usr/bin/mv" 

export dsmc-"${sudo)/usr/bin/dsmc" 

4 check for snapshot name of /dev/null - this indicates that 
4 there is no actual snapshot to be used as in the case of 
4 a hot backup’s archive logs and control file backup, 
if [ "Isnapshot” !- "/dev/null" ]; then 

4 set the "temporary" mountpoint for the snapshots 
tmpmountpoi nt=’7tmp" 
el se tmpmountpoi nt-”" 
fi 

4 we need to mount the backing store file in order to use the snapshot 
// of the filesystem for the backup, the snapshots are to be mounted in 
4 /tmp with the names of the original mount points (ie: the /opt/oracle 
4 snapshot will be mounted - R/D - as /tmp/opt/oracle which will become 
4 the target for the "dsmc archive" task. 

4 _ 

4 in order to mount it we need to cneate the mount points in the /tmp 
4 filesystem, to do this we generate a list of pathnames by doing a 
4 "dirname" on each parent directory (going backwards up the directory 
4 tree) and inserting the (parent) directory name in a list (ie: Inames) 
4 previous to the current directory. 

4 

4 ie: using /opt/oracle "Inames" starts as: "/tmp/opt/oracle", then 
4 becomes "/tmp/opt /tmp/opt/oracle" (should the filesystem name have 
4 more "evels it would have comntinued.) then when names is passed to 
4 the "mkdir" command the parent directories are created at the same 
4 time^tor "just prior" if you like) preventing errors creating a 
4 multi-level mount point name. 

4 this while loop goes on as long as the length of the filesystem name 
4 (ie: I(#fs}) is greater than one (ie: "/"), each time we go through the 
4 loop the tfs variable gets "truncated" by making it equal it’s parent 
4 directory name until we get to the root directory. 

4 initially set names to blank in case any parent process exported one. 
names-"" 

4 check and see if our "temporary" snapshot mountpoint(s) already 

4 exist, if not make them. 

fs-Ifilesystem 

while [ ${#fs} -gt 1 ] 

do 


4 get the name of the backing store file so we can delete it (later) 

# we might want to consider creating the snapshot with the "unlink" 

4 option, rather than getting the path and deleting it later 

4 see the fssnap_ufs marpage 

4 special case exists wth archive logs during HOT backup mode, we do 

# not want to the dsmc archives using the snapshot’ed file system, 

# as they would get stale copies of the data. 

if [ "Isnapshot” !- "/dev/null" ]: then 

backing_store=K!lfssnap} -i -o backing-store Ifilesystem I \ 
cut -f2 -d ’:•) 

logger -ip locall.notice Kbasename 10)": Mounting: ” Isnapshot \ 
"snapshot of:" Ifilesystem 
4 mount the r/o snapshot of our target filesystem 
Ifmount] -F ufs -o ro Isnapshot ittmpmountpoint}!{filesystem} 

fi 

4 Do the actual dsmc archives... (one command per file, for each one in 
4 the filesystem.) Here is where we would change the "dsmc” command to 
4 NetBackup’s "bpbackup" or Legato’s "save" coiriand(s) should we need 

# to use another backup product 
all_rc-0 

while read pathname 
do 

logger -ip locall.notice t(basename 10)": Backing up:" Ipathname 
Kdsmc) archive I {tmpmountpoi ntj I {pathname} -filesonly -archmc-ireten- 
tion_period -desc-tdescription >> Hog 
rc-t? 

logger -ip locall.notice Kbasename $0)": Backup of: " Ipathname ”rc=" Ire 
echo Ipathname "Backup RC-" Ire 
all_rc-I{expr Iall_rc + Ire) 
done < Ipathnames 

4 umount snapshot mount point, delete the snapshot, remove the 
4 backing store file, and finally quit, 

if [ lalljrc -eq 0 ]: then 

If [ "Isnapshot” !- /dev/null ]; then 

logger -ip locall.notice Kbasename 10)": Unmounting:" Isnapshot 
tiumount) ttmpmountpoint/tfilesystem 
4 before deleting capture the usage 

tlfssnapl -i -o snapnumber.blockdevname.rawdevname,mountpoint,state.! 
backing-store.backing-store-len.maxsize.createtime,chunksize \ 

Ifilesystem >> Hog 

$(fssnap} -d Ifilesystem 

logger -ip locall.notice Kbasename I0)" : Snapshot: " Isnapshot 
"deleted" 

Hrm} Ibacking_store Ipathnames 
fi 
fi 

4 We are going to leave the temporary mount points in place, they will be 

4 deleted by a reboot (if they were created in /tmp) 

exit 
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SCRIPTING 


Backup Scripts from 
UnixReview.com, 3rd Edition 

Edited by Ed Schaefer 


I still host the monthly Shell Corner at UnixReview.com. This 
third edition of “Backup Scripts” features scripts from three pil¬ 
lars of the comp.unix.shell newsgroup. If you've posted to that 
newsgroup, Heiner Steven. Michael Wang, or Chris F. A. Johnson 
may have answered your question. The first script described in this 
article creates a local archive of files on a remote host. The second 
script creates a Solaris alternative backup boot disk, and the third 
script automatically extracts files of various archive types into the 
current directory. 


and writes it to standard output. The remote shell copies this output 
from the remote host “darwin” to the local host “newton”, w here I 
set up the copy process to the tape device. 

Although I might have used cat > /dev/rmt/O. some tapes need 
the input in 10-k blocks, so I used: 

dd of=/dev/rmt/O bs-lOk 

which blocks output in exactly the required method. Putting every¬ 
thing together. I get the following (simplified) command line: 


i otwUi%\ 

iD* 

to 




Backup by Heiner Steven 

My script, “Backup” (Listing 1), creates an archive of tiles and 
directories on a remote system, writing them to a local tape drive 
(e.g., a DAT or DLT device). Backup is written in a portable 
Bourne-shell style, which is understood by most derived shells (e.g., 
bash, ksh, and ksh93). With slight modifications, it runs on Solaris 
and Linux. 

For example, assume you have a 
local system (named “newton”), 
which possesses a DAT tape and can 
be used to store backups. Executing 
this command: 

backup darwin 

is sufficient to write a cpio archive 
of all files on the remote host “dar¬ 
win” to the local tape device (e.g., 

/dev/rmt/O). The command: 

backup darwin etc usr/1 oca 1 

saves only the “/etc” and 

“/usr/local” directory hierarchies on ^ ^ 

host “darwin”. Observe that all rela¬ 
tive path names not starting with a 

slash (“/”) are interpreted relative to the remote root (“f) directory. 
So how does it work? Backup uses the remote shell, rsh (on some 
systems called remsh), to run the following commands on the 
remote host: 


rsh darwin ’cd /; find * -xdev -print 
dd of=7dev/rmt/O obs-lOk 


cpio -o’ 







The rsh command functions only if there is an entry in the target sys¬ 
tem's “.rhosts” file allowing the backup script to log-in non-interac- 

tively. Alternatively, ssh can be used 
in the place of rsh (see below). 

The previous command line 
copies remote tiles and directories 
to a local tape. What if I want to 
restore the data? The following 
command line reads a cpio archive 
from a local tape and extracts the 
files and directories on the remote 
system “darwin”: 

dd if=/dev/rmt/0 ibs=10k | 
rsh darwin 'cd /; cpio -1vdu' 

The script can easily be configured 
and modified for different purposes. 
If you prefer the Secure Shell, SSh. 
over rsh, just uncomment the lines 
in the script marked Use ssh. It you 
want to use tar, replace find * - 
print | cpio -o with tar cvf - *. However, I prefer the find 
command's flexibility and file selection options (e.g.. -intime -7, to 
list only files that have been modified in the last 7 days, or - newer 
. ti me Stamp to process only files newer than the file “.timestamp ). 


a\\C® 




cd / 

find * -xdev -print | cpio -o 

The find command creates a list of the local files on host “darwin”. 
-Xdev ensures that find does not traverse non-local (e.g., NFS) 
mounted directories, cpi 0 then reads the list, creates a cpi 0 archive, 
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Copyroot by Michael Wang 

If you manage Solaris systems, you know the importance of the 
boot disk. To protect the boot disk, administrators have opted to use 
online mirrored disks in addition to regular backups. The great advan¬ 
tage of online mirrors is hardware failure protection; w'hen one of the 
mirrored disks fails, the system will still be up and running. 
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PEOPLE ARE TALKING 
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Rackable Systems beats IBM, HP & Dell with lowest Total 
Cost of Ownership for large-scale data center deployments. 


TIRED OF THE OLD WAY? 

A recent cluster server study ranked Rackable Systems #1 in 
lowest Total Cost of Ownership in a head-to-head comparison 
with IBM, HP and Dell. With proof like that, why would anyone 
pay more for the rigid, inflexible and antiquated ways of the Tier 1 
players? It's no surprise that industry leaders Biogen, Electronic 
Arts, Google, Inktomi, Lawrence Livermore National Labs, nVidia, 
Sony America,Tellme Networks, UCSC Human Genome Project, 
Webex and Yahoo! have all deployed Rackable Systems' rack- 
mount server solutions. 


Flexible Configurations 


Rackable 

systems 



- Customized & 
built-to-order 

- Open architecture 

1 Superior remote 
management 


THERE IS A BETTER WAY 

As pioneers of large-scale server deployment solutions, Rackable 
Systems continues to lead the industry with innovative products 
for our customers. Our open architecture system-level configura¬ 
tions are built-to-order to ensure maximum flexibility and ease of 
serviceability. Our patented cabinet-level design features twice 
the density at a fraction of the cost—while optimizing thermal 
effciency and power distribution. And best of all? We can ship your 
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However, if there is a file error, file system corruption, or a 
human error, both of the mirrored disks are at fault. It is slow 
and difficult to restore from backup tape, especially when the 
backup/restore software is unavailable because of an error. 


Listing 1 backup 




mmmHHHHHHHHHmmmmmmmmmmmmmmmwmm 

# Listing 1: 

if Shellscript: backup - backup (remote) system on local tape 

ff Author : Heiner Steven <heiner.steveniodn,de> 

# Requires : cpio, find, {rsh | ssh} 

if Category ; System Administration 

if SCCS-Id. : 8(f) backup 1.1 03/05/29 


# Description 
if 


PN-'basename "JO"’ 
IfER-’l.r 


if Program name 


if Variables written in UPPER CASE can be set in the environment before 
if invoking this script. The following lines set default values if the 
if variables were not set: 


: $[TAPE:=/dev/rmtOj 


if Local tape device 


# Program to run a command remotely. Should have the syntax 
if $RCMD SRCMDFLAGS host command [...] 


: ${RCMDrsh} 

: SfRCHDFLAGS="-l root"} 
if: ${RCMD: =ssh} 

#: S(RCMD:-"-1 root"! 


if Remotely run command 

if Use "ssh" 
if Use "ssh" 


usage 0 { 

echo >42 ”$PN - backup (remote) system on local tape, SVER 
usage; $PN rhost [directory ...] 

The specified directories (default: all) on the host \"rhost\" are 
combined into an V’cpioV archive, which then is written to a local 
tape device ((TAPE)." 


exit 1 




set *' 'getopt :h 
[ $if -It I ] 44 usage 

while [ tif -gt 0 ] 
do 


usage 


if "getopt" detected an error 


case " 

$1" (n 


# your 

--) 

shift; break;; 

-h) 

usage;: 

-*) 

usage;; 

*) 

break;: 

esac 
s h i ft 



done 

[ iif -gt 0 ] || set -- 1 ocalhost 
Rhost=tl; shift 

[ $if -gt 0 ] || set -- ’.[!.]* .' 


if Fi rst host name 

if Default remote host 

if Default remote directory pattern 

if Is STARE a valid device? 


if [ -c "iTAPE" -o -b "STAPE" ] 
t hen 

echo >42 "Write directories from $ Rhost to local device tTAPE ( A d = no)? \c" 
read dummy | { echo >42: exit 0: } 

el se 

echo >42 "cannot access backuo device TAPE-$TAPE" 
exit 1 
fi 

if [ X'IRCMD $ RCMD F LAGS "$Rhost" pwd' !- X ] if Try rsh/ssh 
then 

if Create a remote "cpio" archive, and send it over the net to a 
if local "dd" process, that writes it to the tape device. 

set -x 

"SRCMD” $RCMDFLAGS ARhost" 'cd /; find -xdev -print | 

cpio -ov -H ode' , 
dd of-tTAPE obs-lOk 
set +x 


el se 


echo >42 "$PN: cannot reach host iRhost (.rhosts entry for root?) 
echo >42 " $ p n : Please verify that ViRCMD SRCMDF LAGS $ Rhost pwd\" works’ 
exit 1 


fi 


I introduce a simple, different approach, which may be used 
instead of. or in addition to. online mirrors. Copyroot (Listing 2) 
creates a Solaris alternative backup boot disk. Rather than creating 
an online mirror, copyroot makes a delayed, ‘'offline * mirror. 
Changes on the primary disk do not immediately propagate to the 
backup boot disk, providing a chance to correct the error, or fall 
back to the last known good OS. 

The backup boot disk does not prevent the server from going 
down. When the primary boot disk crashes, however, you can sim¬ 
ply type a hoot command to boot from the backup disk. You can 
also automatically boot when the “boot-device" is set to multiple 
boot disks. 

Solaris allows you to define a device alias using "nvalias", for 
example: 

nvalias bkupboot /pci@1f,4000/scsi@3/disk@8,0 
The above alias allows this boot command: 
boot bkupboot 

which boots the system from the backup boot disk. The boot-device 
can he defined to have multiple boot disks with, for example: 

ok> setenv boot-device pnmboot bkupboot 

Whenever a server needs to be shut down for maintenance tasks, I 
always perform the ritual of boot bkupboot, shutdown, boot to ver¬ 
ify the backup boot disk. Also, you do not need to choose between 
using online mirrors and the backup boot disk: it you can afford a 
third disk, you can have both. 

Copyroot was written in the Solaris 2.3 days, and works all the 
way up to Solaris 9, I recently updated it to support a configuration 
where the primary boot disk is mirrored using Solaris Volume 
Manager, which is part of Solaris 9. 

I did not test copyroot for creating a boot disk under Veritas 
Volume Manager, but it should work. However, one of the Sun 
BluePrints papers (see References) “strongly discourages" Veritas 
Volume Manager on boot disk in favor of Solaris Volume Manager, 
citing that “VERITAS volumes do not. by default, correspond to 
partitions”, and mentions “extra complications”. 

Now, let’s review how copyroot works, following the program 
flow, and emphasizing the major points. 

Setting Up Working Environment 

The shell used is ksh93d, which is part of Solaris distribution at 
least since Solaris 2.6. During the recent update, some new features 
of ksh93 came in handy. Following shell definition. T always define 
PATH, which solves the “can run on command line, but not on cron" 
problem, bull tin -d uname removes uname as a ksh93d built-in 
command, forcing uname to be /bin/uname. The built-in uname -i, 
which previously delivered unexpected results, is fixed in the recent 
ksh93 shell. 

Lines 13-24 are generic: they set up the OPT (“OPT Power 
Tools”) directory structure, one of my earlier designs: 

* All jobs reside in a common directory ($my_dir). 

• The configuration files, or associated programs, if any. are in the 
same directory as jobs, and can he derived from the job names 
($my_name). For example, the configuration corresponding to 
$my_name would be $my_name.cf. 
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• The log directory Smyjogdir for each job is 
$ray_dir/log/$my_name. 

• Log file (Smyjogfile) has the time__stainp appended, and the 
alphabetic order {Is listing) corresponding to the time order. 


• I redirect all the standard error and output to the log file, which 
catches all the errors —especially unexpected errors. 

• Log tiles are cleaned by the program itself. The retention period is 
defined in the configuration file, or set to the default value. 
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#! /usr/dt/bin/dtksh 

# 

if Listing 2: copyroot 
if Author: Michael Wang 

|| This script creates a backup boot disk from primary, 

f T ^ s script was based on an idea by co-worker Dave Singer circa 1996 
j Both of us worked for Lucent, then AT&T Sell Labs, in Whippany, Nl! 
f Dave originally used the dd command but I rewrote it using ufsdump and 
it ufsrestore, and updatec for Solaris Volume Manager. 

PATH=/usr/xpg4/bin:/usr/bin:/usr/sbin 
builtin *d uname 

my_name-${0#*/} 

my_dir=$(0S/*l 

myjogdi r-Jmy_di r/log/$my_name 

[[ -d imyj ogdi r ]] || mkdi r -p tmy_logdir 

W J ogfi 1 e=$my_l ogdi r/$my_name. $ (da t e +’lm-*d JH: £M") 

exec l>$my_logfile 2>&1 

date 

my_conf={my_dir/$my_name,cf 
[[ -r $my_conf ]] | { 

print "The configuration file (tmy_conf! does not exists " 
exit 10 

} 


. tmy^conf 

p_boot-$(print ip_boot) 
bJobt~$( print tb_boot) 
: S{log_keep:-30) 
find Jmyjogdir -type f 


-mtime +$1og_keep -print -exec rm {} \ ; 


[[ "fiuname -n)" — "$machine_name" ]] || { 
print This script should be run on Smachime name only 
exit 20 - j. 

1 


pipe-”I” 

mount 1 egrep "Vcopyroot | on (${b_boot/7 /Spipe} > 
print "/copyroot or bacxup disk mounted already " 
exit 30 
J 


&& { 


do 


set -A pa -- Sp_boot 

for (( slice = 0; slice <= ${#pa[@]J-l ; slices )). 

(( siice — 1 )) && continue 
[[ -n tbackup_dir ]] && dump_fi1e-Jbackup_dir/tslice.dump II 

dump_file-"/dev/null" 

ufsdump Of $dump_fi1e $(pa[$sTicej} 2>&i < /dev/null j while read j; do 
print r tj j egrep \ 

Writing [0-9.]+ kilobyte recordsj\ 

Date of {this|last) level 0 dump: |\ 

Dumping /dev/.* \(.*\) to|\ 

(Mapping|Dumping) \(Pass ,*\) \[{regular files|directories)\]|\ 
Estimated [0-9.]+ blocks \C[0-9.]+(M3JKB)\) |\ 


DUMP 

DUMP 

DUMP 

DUMP 

DUMP 

DUMP 

DUMP 

DUMP 


[0-9.]+ blocks \([0*9.]+(MB|KB)\) on .* at|\ 

[0-9,]+SS done, finished in|\ 

DUMP IS DONE" && continue 
pnint -r -- "ij" 

print "Primary root filesystem has problems " 
exit 40 
done 
done 

prtvtoc /dev/rdsk/cQtOdOsO | fmthard -s ■ /dev/rdsk/c0t9d0s0 

set -A ba -- ib_boot 

[[ -d' /copyroot ]] II mkdir /copyroot 
tab-i(print "\t") 
cd / 


do 


for (( sl^ce = 0; slice <= if{/ba[@])-l ; slice++ }) 
(! slice = 1 )) && continue 
print y | newfs i(ba[$slice]J 
fsck -y t[ba[Sslice]J 


mount $iba[$s1ice]} /copyroot 

df -Pk /copyroot j grep B *J[ba[*sl1ce]} " II exit 60 
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if [[ $dump_file -= "/dev/null" ]]; then 
ufsdump Of - ${pa[$slice]} | ( cd /copyroot && ufsrestore rf - } 
else 


fi 


( cd /copyroot && ufsrestore rf Jbackupjir/Ssl ice.dump ) 


;( slice == 0 }) && { 
for (( s=0; s <- t(#pa[8].J-l; s++ )); do 
pd-Sfpa[$s]} bd=$[ba[ts]} pr=tfpd/dsk/rdsk] bi—$fbd/dsk/rdskI 
sed "s:$pd\([ $tab]\):ibd\l: : 
s:tpr\([ itab]\):tbr\l;; 

/copyroot/etc/vfstab > /copyroot/etc/vfstab.tmp 
op /copyroot/etc/vfstab.tmp /copyroot/etc/vfstab 
done 

grep "Vootdev:" /etc/system && [ 


1 


} 


sed / A rootdev:/s: A :+ /etc/system > /copy root/etc/system 


umount /copyroot 
fsck -y $ {ba[$slice]} 

106 done 

107 

108 install boot /usr/platform/Kuname -i >/l i b/fs/ufs/bootbl k $lba[0]/dsk/rdsk] 

110 rmdir /copyroot 

111 date 

112 print "copyroot finished on J(uname -n) successfully " 

113 exit 0 
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Sourcing the Configuration File 

Copy root needs some information to start, which is obtained by 
sourcing the configuration file, copyroot.cf. Here is a configuration 
file sample: 


machine_name-"ziqi" 
p_boot=$( 

print /dev/dsk/cOtOdOsO 
print /dev/dsk/cOtOdOsl 
print /dev/dsk/c0t0d0s3 
print /dev/dsk/c0t0d0s4 
print /dev/dsk/c0t0d0s6 

) 

b_boot=t( 

print Zdev/dsk/cOtSdOsO 
print /dev/dsk/c0t8d0sl 
print /dev/dsk/c0t8d0s3 
print /dev/dsk/c0t8d0s4 
print /dev/dsk/c0t8d0s6 

) 

backup_d1r-7apps2" # optional directory to hold boot disk backup 


The variables defined in the configuration tile are: 

• Define the machine_name. The purpose is to compare the defini¬ 
tion with output from Lina me -n, making sure copyroot runs on the 
correct machine. 


Listing 3 unpack 


#!/bin/sh 

# Listing 3: 

# Fri Jan 5 14:57:59 EST 3001 

# /usr/local/bin/unpack 

# Copyright 2001. 2002, 2003, Chris F.A. Johnson 

# Released under the terms of the GNU General Public License 

for file: do 

case if ile in 

*.zip]*.ZIP) 

unzip "tfile” 


*.tar) 

tar xvpf "Ifile" 


# ,tgz and .tar.tgz are equivalent: tar 4 gzip; 

## .1 (UNIX compress format) can also be uncompressed by gunzip 
*.tgz|*.tar,gz|*.tar. 1) 

gunzip -c "ifile" | tar xvp 


# tar & bzip2 
*.tar.bz2|*.tbz2) 

bunzip2 -c "ifile" | tar xvp 

# gzipped and bzip2ed files are uncompressed to the current 

## directory, leaving the original files in place 

*.gz) 

gunzip -c "ifile" > "'basename "ifile" ,gz ” 


## compress 
*.Z) 

gzip "C "ifile" S basename "ifile" . Z 


H bzipZ 
*.bz2) 

bunzip2 *c "ifile" > "'basename "ifile" ,bz2 " 

esac 

done 


• Define the device names for the primary boot disk, and define the 
backup boot disk. Copyroot assumes that the first item in the list 
is the device name for the root file system, which every Solaris 
sytem must have. The second item is the dedicated swap partition. 

If you do not have one. or if it is not on the boot disk, use as a 
placeholder. The primary device names and backup device names 
should correspond one-to-one. 

• Reverse copy is done simply by swapping the definition of p_boot 
and b_boot. This is needed when the primary boot disk crashes 
and is replaced. 

• "backup_dir” is an optional directory to hold the boot disk 
backup, which I will clarify later. 

When the boot disk is under the Solaris Volume Manager, the defin¬ 
ition of p_boot looks like this: 

p_boot“l( 

print /dev/md/dsk/dO 
print /dev/md/dsk/dl 
print /dev/md/dsk/d3 
print /dev/md/dsk/d4 
print /dev/md/dsk/d6 

) 

Checking the Run Machine 

Ensure the script executes only on the box where you want it to 
run. To disable the program, simply define machine_name as some¬ 
thing that does not match uname -n. This might be an option, for 
example, when the OS on the primary boot disk is upgraded, and 
you want the backup boot disk to still have the old OS for an 
extended period of time (see lines 37-40). 

This erroring out also reminds you to change the maehine_name 
back to uname -n, after you feel the upgrade is okay. 

Checking the Mounted File Systems 

To perform duplication using ufsdump and ufsrestore, mount the 
backup disk slices under /copyroot. Initially, nothing should be 
mounted under /copyroot. If there is, perhaps something else is 
mounted. Also, the backup disk slices should not be mounted at all. 
If these conditions happen, something is wrong and copyroot should 
not continue (see lines 42-46). 

Performing the Dry Run 

Lines 48-67 perform the dry run. Copyroot cycles through each 
slice except the swap slice, running ufsdump on each slice, and out- 
putting to /dev/null, or a backup file under Sbackup_dir if it is 
defined in the configuration file. 

Copyroot uses ufsdump to check the primary boot disk. If the 
primary disk has a corruption, it most likely shows up in a dry run. I 
then terminate the program, preventing the corruption to propagate 
to the backup disk. This is not a perceived caution, but a lesson 
learned the hard way. 

Since 1 do not know what error messages ufsdump may produce. 
I parse the messages, line by line, checking for known, expected 
output. Anything other than expected output is considered an error, 
and the program terminates immediately. This is more reliable than 
checking the exit code. 

Saving the ufsdump file, and using it later on for restore not only 
avoids the need to read the boot disk twice, but also is safer because 
the restore uses the verified backup, instead of ufsdump output on 
the fly. 


# for verification purpose 

# primary boot disk partitions 


# backup boot disk partitions 
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Partitioning Backup Disk 

A prerequisite of using copy root is partitioning the backup disk, 
which needs to be partitioned only initially, and in the rare case 
when the primary disk is repartitioned. If the backup disk is of the 
same size as the primary, you can. and should, partition the backup 
disk as the primary. The syntax is shown in a comment (line 69) or 
you can do it manually. 

Copyroot, which uses ufsdump and ufsrestore, does not require 
the backup disk be the same size or have the same partitions as the 
primary. It only requires the backup partitions be large enough to 
hold the same amount of data as the primary. 

Performing the Actual Run 

Lines 71-106 create the backup disk. For each slice of the 
backup disk except the swap partition, execute newfs to make a new 
UFS file system, and check it with fsck (lines 77-79). The fsck may 
not be necessary for newly created file systems, but won't hurt. 
Next, mount the slice and verify the mount directly (lines 81-82), I 
ufsrestore every slice, skipping the swap partition, from the primary 
to the backup, using ufsdump output on the fly or the saved ufs¬ 
dump file (lines 84-88). 

The root file system needs special attention as follows. The 
vfstab file contains the boot disk device name for boot strapping 
purposes. The file on the backup boot should contain the device 
name of the backup disk instead of the primary. Code lines 91-97 
perform the changes. 

If the primary boot disk is mirrored using Solaris Volume 
Manager, the /etc/system file contains a line added by "metaroot”; 

rootdev:/pseudo/md@0:0,0,bl k 

which defines the root device to be the mirrored volume. This line 
needs to be commented out in the backup /etc/system (lines 99-101). 

Each slice is synchronized from the primary, /copyroot is 
umounted, and the partition is fscked again —just to be sure. 

On a Sun E250 with 2x30GMHz CPU, 896 MB memory, all 
SEAGATE ST318405LC disks, running Solaris 9 full installation, 
copyroot requires about 12 minutes to back up the primary disk to 
the $backup_dir on the third disk, and restore to the backup boot 
disk, Although relatively short, the synchronization process is inef¬ 
ficient and vulnerable. Considering most of the data is static. I’m 
seeking a better method. 

Install the Boot Block 

Last, we install the boot block, otherwise it won't boot (line 108), 
(For those familiar with Linux, this is analogous to lilo, or grub). 

In conclusion, copyroot creates a backup boot disk using a sim¬ 
ple, standard method, which provides protection for the primary 
disk data. In case of hardware failure, copyroot reduces downtime 
by booting from the backup boot disk. It can be used with Volume 
Manager mirrored boot disk to provide both online availability and 
data protection. Future versions of copyroot, if any, will be found at 
http://www.unixlabplus.com/unix-prog/copy root. 
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Unpack by Chris F. A. Johnson 

Backups come in all shapes and sizes, created on many different 
platforms with many different archiving tools and compressors. To 
extract the files, there are just as many different utilities, though 
some can handle more than one type of archive. To make life more 
interesting, the same type of archive file may have different suffixes 
(e.g., .lar.gz and .tgz are the same thing). 

While it's not difficult to remember which archive utility goes 
with which type of file, or even to remember which is or is not 
installed on the current system, why not let the computer do it for 
you? I wrote a short script to do just that. The result is unpack. Now. 
whatever the format. I just type "unpack FILENAME.WHAT¬ 
EVER" and the files are extracted under the current directory. 

Any number of archives files can be specified on the command 
line, and they don’t all have to be the same type — you can mix zip 
files with gzip files with bzip2 files with whatever other format you 
have. 

The script is very simple — a case statement within a "for" loop. 
I don't remember ever using more than a single file on the command 
line, but the script supports more. Each type of file is recognized by 
case’s pattern-matching capability; different forms for the same 
type of file are handled on the same line. 

The script is written to be as portable as possible. To that end, it 
does not use the "z" option to tar, since not all versions of the pro¬ 
gram include it. Ditto for "j". and its older equivalent, "y". Instead, 
gunzip (or bunz.ip2) is called w r ith the "-c" option and the output 
piped through a generic tar command. 

There are many enhancements that could be made to unpack, 
including options to unpack in a different directory, to select silent 
or verbose operation, or to customize error messages when a file is 
not found or a program such as bzip2 is not installed. However. I'm 
not likely to bother with those unless, of course, a situation arises 
w'here l really need such a feature. 

The only enhancement I am likely to add is other archive for¬ 
mats, such as cpio, pax. and perhaps various Amiga archive formats, 
it you want to add something to unpack, go right ahead (and please 
send me a copy). 
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T hree years ago. the U.S. Patent Office (www.uspto.gov) had 
a major performance problem in a critical application. A key 
search system was suffering major performance problems, 
yet IT technicians needed a method to model and design specific 
scenarios to optimize the proposed solution. The technicians contacted 
Hewlett-Packard, maker of the Patent Office’s HP Measure Ware 
performance management software, and an HP contact suggested that 
the TeamQuest Model performance-mode ling tool could save valuable 
time in optimizing the proposed solution. 

By employing analytic modeling and validating its search system 
application using Model, the Patent Office found the I/O bottleneck 
immediately. After locating the problem, technicians used Model to 
help test a variety of channel “striping” sizes and find the appropriate 
Fibre Channel configuration. Model enabled the Patent Office to 
aggregate statistics for each storage drive across the Fibre Channel, 
making it possible to identify bottlenecks and then determine the best 
method for dividing 1/0 across the channel to eliminate those 
bottlenecks. 

Today, the Patent Office is using Model regularly to import, analyze, 
and enhance data from Measure Ware. Model captures workload data 
from existing systems, and applies that data to the Patent Office's new' 
system model, projecting workload growth over the lifespan of the 


IT executives at the Patent Office say 
that TeamQuest® Model has saved 
them millions of dollars in over¬ 
processing costs 


server. This approach enables the Patent Office to avoid over¬ 
provisioning over the life of the system, because the IT organization 
can accurately predict the need for future capacity. 

And what of the Patent Office’s search system? In early 2000. the 
Patent Office was awarded a Department of Commerce Silver Medal 
Award for the Patent Search Performance Team’s improvement of its 
multimillion-dollar search engine project. IT executives at the Patent 
Office say that TeamQuest Model has saved them millions of dollars in 
over-provisioning costs, and helped make the search system project a 
“roaring success.” 



EMA Perspective 

In a difficult economy, every IT dollar matters. The days of solving 
performance problems by indiscriminately buying more servers are 
over, and there is an excellent window of opportunity for deploying 
management technology that can optimize server utilization and 
performance. IT organizations are looking for ways to make better use 
of existing server capacity while reducing the need to buy more 
processor power. 

At the same time, u is important to note that IT budgets are tight, and 
that most IT organizations are not prepared to purchase new 
management or planning software unless they are guaranteed a clear 
and fast return on their investment (ROI). Today's management tools 
must be affordable, easy to deploy, and show strong results in a very 
short period of time. 


Through capacity planning IT 
organizations can realize major cost 
savings immediately 


Capacity planning tools meet all of these criteria. Through capacity 
planning, IT organizations can realize major cost savings immediately 
by reducing or eliminating the need to over-provision their server 
environments. Capacity planning and modeling technology also can 
guide IT through the server consolidation process, and can help set 
realistic goals for service levels that can be maintained over time. 


Copyriaht © 2003 TeamQuest Corporation. All Right* Reserved. TeamQuest and the TeamQuest logo are registered trademarks in the US. EU and 
elsewhere. All oilier trademarks and service marks are the property of their respective owners. 
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Domain Management Using LDAP 

Divya Sundaram 


I f you're a systems administrator for a large network, you have to Donley. I use the PerLDAP modules from Mozilla. but the others 

manage a wealth ot information to keep the network running effi- are also quite adequate for most systems administration tasks, 

ciently. Where is a server located, w hat rack, who owns it. when 



was it last patched, etc.? Sometimes we squirrel away data like that by 
including comments in the hosts map or in a knowledge database. 
Some organizations spend a tremendous amount of time and money 
developing or adapting asset manage¬ 
ment or change management soft¬ 
ware to track this information. 

I adapted the LDAP server used 
for Unix naming services to manage 
all this information. By deploying 
an LDAP directory with a schema 
that conforms to RFC 2307. you can 
relatively easily transition to using 
LDAP as a naming service instead 
of NIS or even YP. 

When we deployed the Sun ONE 
Directory Server as a component of 
Sun Microsystems' Solaris 9. it 
opened up some new opportunities 
to exploit LDAP Directories for 
managing the servers and services 
in our Unix domain. Having a well- 
designed and maintained LDAP 
Directory represented an opportunity to realize significant cost sav¬ 
ings by replacing expensive and proprietary solutions for managing 
domain and server information and for asset management. 


The RFC 2307 Based LDAP Directory Service 

The LDAP Directory Server is an object database capable of manag¬ 
ing millions ot entries. This means that, instead of the traditional key- 

value pairings used for traditional 
Unix maps, there is now the primary 
key and an arbitrary number of 
attribute values available as part of the 
map entry. The number and types of 
these attributes are essentially indefi¬ 
nitely extensible and allow the LDAP 
database to be used to replace or aug¬ 
ment other domain management tools. 

RFC 2307 lays out a method for 
storing the standard NIS maps into 
LDAP directories. It defines a 
schema for an LDAP database and a 
method for mapping NIS map 
entries into objects in the LDAP 
directory. This RFC has been 
adopted by most of the major Unix 
vendors. You can read the RFC at: 

http://www.faqs.org/rfcs/rfc2307.htinl 
For example, the following hosts map entry: 


What Software You Need 

To begin, you need an LDAP server. 1 use the Sun ONE Directory 
Scrvei lor all the examples in this article, because it's bundled with 
Solaris and is what I use at work. You could use Novell’s eDirectorv. 
OpenLDAP. or even the Microsoft Active Directory. 

You also need the C LDAP SDK. You can use Mozilla's. or you 
can download one from Netscape or Sun. You can also use the soft¬ 
ware from the OpenLDAP Project at: 

http://www.openldap.org 

The SDK comes with two tools that are especially useful: 
ldapsearch and ldapmodify. These are respectively used to query 
and update an LDAP database. I chose to use these over the ones 
that are shipped with the LDAP server because they can be statically 
compiled and installed on all Unix systems. 

Most of my scripts are written using Perl. There are two options 
for writing Perl scripts that interface with LDAP: PerLDAP from 
Mozilla and Net::LDAP (or. confusingly. Perl-ldap) by Graham 
Ban-. There is also the now obsolete Net::LDAPapi by Clayton 
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192.168.0.201 so!91oginl # Solaris 9 Login Host 
becomes: 

* /opt/myOrg/bin/ldapsearch -h ldapO.avnika.com -b \ 

"ou-hosts, dc-avnika, dc=com” cn-=sol91 oginl 

dn: cn-sol91ogin1+1pHostNumber-192.168.0.201, ou-Hosts. dc-avnika. dc-com 

objectClass: ipHost 

objectClass: device 

objectClass: top 

cn: sol91oginl 

ipHostNumber: 192.168.0.201 

The comment information contained in the hosts map file was dis¬ 
carded. The other NIS maps are similarly mapped into LDAP data¬ 
base objects. 

We have a large netw-ork consisting of servers used by various 
development groups, as well as servers dedicated to running busi¬ 
ness applications. We have a series of machines with the Sun ONE 
Directory Server 5.2 deployed on them. Most of the LDAP servers 
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are multi-homed and strategically located across the network so that 
most NIS traffic does not leave the subnet of the NIS clients. This 
makes the LDAP directory a highly available and. through the use 
of replicas, robust and distributed repository' for storing this data. 

Schema Extensions 

When extending schemas, it is important to ensure that the object- 
classes and attributes are named uniquely so that they will not clash 
with the naming of the attributes and objectclass by other tools and 
applications that may be sharing the LDAP directory'. Here I used the 
prefix “myOrg” for all of the attributes I added — you can. and should, 
substitute the appropriate name or acronym that uniquely describes 
your organization to try' to minimize the chances ot naming collisions. 

I first created an objectclass "myOrgHosf ’ that would contain all 
the attributes 1 have added to the schema. For the Sun ONE 
Directory Server, objectclasses and attributes can be added from the 
Sun Console. This is shown in the dialog depicted in Figure 1. If I 
had chosen either the objectclass ipHost or Device as the parent 
objectclass, the attributes could only be associated with hosts. I 
chose to derive it from the objectclass top so that the LDAP data¬ 
base could be extended to track items that are not hosts like 
RAID arrays, Brocade switches for the LANs, etc. 

Once the Objectclass has been created, another dialog is used to 
add attributes to this objectclass. Figure 2 shows the dialog used to 
add a given attribute. All attributes are created as “Case Insensitive 
String" type (also known as “DirectoryString"). I have made most 
of them multi-valued — except where having multiple values does 
not make sense. For example, the attribute myOrgDefaultRouter is 
single-valued, because it does not make sense to have multiple 


Figure 1 Creating an Objectclass using the Sun 
Directory Server Console 
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default routers. Making these single-valued also means that any 
operation that tries to make them multi-valued will fail. 

Asset Tracking Using the Hosts Map 

Because every host in the hosts map will be represented in the 
LDAP directory, it makes sense to extend that map to track information 
that may otherwise be managed in asset tracking systems. The chal¬ 
lenge is to decide what to track in the database and w'hat to leave out. I 
was able to capture information based upon what I needed to track: 

dn: cn—sol91oginl+ipHostNunber=192.168.0.201, oncosts, dc-avnika, dc-com 

objectclass: ipHost 

objectclass: device 

objectclass: top 

objectclass: myOrgHost 

cn: sol91oglnl 

ipHostNumber: 192.168.0.201 

myOrgDefaultRouter: 192.168.0.254 

myOrgTriJack: 1-22*34 

myOrglriJack: 1-22-35 

myOrgTriJack: 1-23-08 

myOrgTriJack: 1-23-09 

myOrgRootMail: divya.sundaram@avnika.com 

myOrgTerminalServer: mwtermserverl.avnika.com:14 

myOrgServerLocation: 4445-12-5 

myOrgFacility: MIDWEST 

myOrgCustomer: Compiler Development Group (CDG01) 

myOrgCustomer: Compiler QA Group (CDG05) 

myOrgCustomer: Architecture and New Projects Group (CDG67) 

myOrgSlA: Level A 

myOrgAssetNumber: sun-6CEDFGHV 

myOrgServerType: Sun E 450 

myOrgServerMemory: 2GB 

myOrgServerNumCPU: 4x450Mhz 

In this example, I have taken the standard entry' and added other 
attributes to capture information about the host that l want to track. 
Some attributes are multi-valued: servers that are either multi-homed 
or have trunked connections will be "punched down to multiple 
locations as shown by the myOrgTriJack attribute. 

When troubleshooting this host, 1 can immediately discern a lot ot 
information — its location (myOrgServerLocation). who’s going to be 
primarily impacted by an outage on this server (myOrgCustomer), and 


Figure 2 Creating an attribute using the Sun Directory 
Server Console 
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who I should contact as the primary administrator for this server 
(my OrgRootMai 1). The SLA level (myOrgSLA) actually allows me 
see the level of service associated with this host and helps determine 
the escalation process to be followed in resolving issues with this host. 

The schema can be extended to house other host-specific infor¬ 
mation that may usually be tracked in Asset Tracking software as 
well. Although LDAP directories don't offer a syntax as rich as 
SQL's for queries that are used in generating audit reports. 1 have 
found that scripts using PerLDAP (http://www.perldap.org) are 
quite adequate for generating snapshots for audit reports. 

Using PerLDAP, it was also extremely simple to build CGIs that 
updated the data in the LDAP database. A host configuration Web 
page allows administrators to update the information associated 
with a given host. The CGI also performs some error checking to 
ensure that the information is not duplicated or in error. For exam¬ 
ple, it ensures that a host will not be assigned an IP that is already 
taken by another server, and that the default router listed actually 
exists and is appropriate. 

Server Configuration Tracking Using the Hosts Map 

The second type of information that can be tracked pertains to 
the host's operating system configuration. Here is what I added for 
tracking operating system configuration: 

dn: cn-sol91 ogi'nl+ipHostNumber-192.168.0.201, oncosts, dc=avnika, dc=com 

myOrgETCRelease: Solaris 8 10/00 s28s_u2wos_l1b SPARC 
myOrgPatchLevel: Solaris 8 SPARC May 2003 (applied 2003-07-03) 
myOrgResolvCortf: SearchList: avnika.com 
myOrgResolvConf: SearchList: engineering.avnika 
myOrgResolvConf: NameServer: 192.168.0.10 
myQrgResolvConf: NameServer: 192.168.0.11 
myOrgResolvConf: NameServer: 192.168.0.12 
myOrgResolvConf: Domain: avnika.com 
myOrgNSSWITCH: hosts: files dns Idap 
myOrgNSSWITCH: passwd: files Idap 
myOrgNSSWITCH: shadow: files 
myOrgNSSWITCH: group: files Idap 
myOrgNSSWITCH: bootparams: files 
myOrgNSSWITCH: ethers: files 
myOrgNSSWITCH: netmasks: files 
myOrgNSSWITCH: networks: files 
myOrgNSSWITCH: protocols: files 
myOrgNSSWITCH: rpc: files 
myOrgNSSWITCH: services: files 
myOrgNSSWITCH: netgroup: files 
myOrgNSSWITCH: publickey: files 
myOrgNSSWITCH: automount: files 
myOrgNSSWITCH: aliases: files 

I started off storing the contents of 
/etc/defaultrouter in the host map. Then I 
added the contents of the /etc/release file as 
well as the contents of the resolver configu¬ 
ration (/etc/resolv.conf) and the name ser¬ 
vice configuration (/etc/nsswitch.conf) fdes. 

I also added an attribute (myOrgPatchLevel) 
that captures the latest patch level applied 
to the host as well as the date that it was 
applied. 

On some machines on which I had 
Veritas installed. 1 included the Veritas 


software information and the license keys being used. This can be 
extended to include the software information for other tools that are 
installed on the host. 

For Solaris hosts, one option I considered was to include the 
packages that have been installed on the host as well as all the 
patches or patch clusters that have been applied. 1 do not believe that 
putting all of this information in the Directory directly is appropri¬ 
ate — mainly because I see 580 system packages, 66 application 
and user packages, and 18 other packages installed on my Solaris 8 
servers. This much data can only be tracked by adhering to a 
process for keeping this information current, because the informa¬ 
tion is dynamic. 

For a look into what else you could stuff into the directory, con¬ 
sider the output of the “explorer” tool on a Solaris host. There is far 
too much information there to be stored en masse in the directory. But 
you can pick out the information you consider vital and store that in 
the directory. For example, you may want to store the contents of the 
/etc/vfstab (or/etc/fstab) file. Or. you might choose the active lines of 
/etc/inetd.conf. In essence, anything that is relatively static and is use¬ 
ful to have available when troubleshooting a problem. 

My choices so lar allow me to use the LDAP directory as a 
distributed database containing the configuration information 
necessary to bring any host back online in case of a failure. This 
greatly reduces the cycle time for bringing a server back online 
alter a crash. I can also, at a glance, see what configuration a host 
is supposed to use and whether it deviates from that. Remember 
that the LDAP directory is being used as an NIS replacement, so 
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this information is replicated onto every machine that would 
function as an “NIS slave" in your environment. 

Extending the Host Schema for Account Provisioning 

Our business application servers are deployed to use a sparse 
Passwd file — this contains the system accounts, the application 
account, and the accounts of the application administrators taken 
from the standard Passwd map. All their data is kept in flat files; 
they do not use NIS or YP. For these hosts. 1 keep the flat files 
synchronized with the NIS domain by using scripts to grab the 
information from LDAP and update the various maps that are 
needed. 

As an example, let’s consider an Oracle database server. A Unix 
user, “oracle”, is created and it's the only Unix account that resides 
on the box. The user password is known to the Oracle DBAs and is 
only used in disaster recovery scenarios. Most of the time, the 
DBAs log into the server with their own logins and then use a tool 
like PowerBroker to switch user to "oracle” to perform database 
maintenance activities. 

It is desirable for us to keep the username, userids, and pass¬ 
words consistent between the application hosts and the rest of the 
NIS maps. The alternative is to create point solutions, which 
increases the possibility of confusion and thus the support costs of 
the environment. 

We can do this by further extending the Hosts map to include the 
user information for the host. The second step is to classify hosts 
into functional groups and then use LDAP groups to manage 
account provisioning. 

I will illustrate this using the ease of the Oracle server. We have 
been told that, in addition to the standard system accounts in 
/etc/passwd, we need to provide access to the following users: ora¬ 
cle, divya, samson. rodney, monitor. We can simply indicate this 
information by using an attribute that is added to the hosts entry: 

$ /opt/myOrg/bin/ldapsearch -h ldapO.avnika.com -b \ 

"ou=Hosts, dc=avnika, dc=com" cn=solSoradb 1 

dn: cn=sol8oradbl+ipHostNuinber*=192.163.0.191, ou=*Hosts, dc=avnika, dc=coim 

objectClass: ipHost 

objectClass: device 

objectClass: top 

cn: solSoradbl 

ipHostNumber: 192.168.0.191 

myOrgServiceUserlD: oracle | x 11981881 Ora cl e 

User| /export/oracle| /bin/ksti 

myOrgServiceUserlD: divya 

myQrgServiceUserlD: samson 

myOrgServiceUserlD: rodney 

myOrgServiceUserlD: monitor 

In this case, the user “oracle” will have only its password plucked 
out of the “passwd” map stored in the directory. The rest of the 
information is contained in the attribute as defined above and serves 
to override the information in the LDAP-based “passwd map 
stored in the “ou=people, dc=avnika, dc=com container. 

Meanwhile, to implement the configuration that has been 
applied, on the server solBoradb 1, we use a PerLDAP script to grab 
the information from the LDAP server and write it to 
/etc/passwd. new. Then a cron job swaps out /etc/passwd for 
/etc/passwd.new after performing a sanity check. 

This approach is much like what is done using the old “Yellow 
Paces" NIS (i.e.. yp) where a cron job would be used to download 
the maps onto the NIS slaves. The advantage here is that the script 


can he enabled to permit some sanity checking functions to ensure 
that a corrupt file is not being generated. 

A Web interface can be built to allow administrators to manage 
the information for these sparse passwd maps. However, this infor¬ 
mation must be tracked accurately for every host. Whenever a new 
person joins the support team, the administrators must manually add 
this user’s login ID to the entry for each host that is aflected. 
Because this is a manual process, it is prone to error. This approach 
is best for a few systems, but doesn't scale well when you have a 
large number of servers. 

In our data center, business applications servers may be 
deployed as part of a project, or based upon functionality. For 
example, we have one Oracle Database Administrator Team but 
more than a hundred Oracle database servers. In this scenario, all 
the members of the DBA team would need to have access to all the 
servers classified as Oracle servers. To solve this problem, I created 
an “ LDAP group” in a new branch of the LDAP directory and asso¬ 
ciated hosts and accounts with it. The logical way to do this is as 
follows: 

dn: cn=0racle DBA, ou=application groups, dc=avnika, dc-com 
objectClass: top 

objectClass: myOrgApplicationGroup 
myOrgHostCN: cn=sol8oradbl+ipHostNumber-192.168.0.191 
myOrgHostCN: cn-sol8oradb2+ipHostNumber=l92.168.0.195 
myOrgHostCN: cn=sol7oradb3+tpHostMumber-192.168.0.198 
myOrgHostCN: cn=so!8oradb4+ipHostNumber=192.168,0.212 
myOrgHostCN: cn=sol9oradb5+ipHostNumber=192.168.0.211 
myOrgServiceUserlD: uid=oracle | x| 198188 1 Oracle User I,/export/ \ 
oracle|/bin/ksh 
myOrgServiceUserlD: uid^divya 
myOrgServiceUserlD: uid-samson 
myOrgServiceUserlD: uid=rodney 
myOrgServiceUserlD: uid=monitor 

This entry in the LDAP database indicates both the hosts that are 
classified as belonging to the Oracle DBA team as well as the non¬ 
system accounts that need to be included in its /etc/passwd file. The 
details of the Oracle application account are listed in full, but the 
rest of the user accounts are read from the user s LDAP entry. 

The script that retrieves the information from the LDAP Server 
and parses it to build the new /etc/passwd tile would be installed on 
all the servers and will allow the servers to automatically generate 
the /etc/passwd and /etc/shadow tiles. It could be run periodically 
from within cron. This approach requires some planning, but the 
advantage is that it reduces the possibility of error because updates 
made once get propagated everywhere. 

Conclusion 

Over the next few years, Unix administrators will become 
extremely familiar with LDAP. Sun s decision to include iheii 
Directory Server as part of the Solaris operating system will allow 
systems administrators to leverage the directory for a variety of 
tasks. The ability to create new fields in the LDAP database pro¬ 
vides a great deal of flexibility to solve common systems adminis¬ 
tration problems. 

Divya Sundaram is currently employed as the Manager of Directory 
Sen ices at Motorola. Divya has been a Unix systems administrator since 
1994 in various roles at Motorola and been involved with LDAP directories 
since 1997. 
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Questions and Answers 

Amy Rich 


Q l’m on a Solaris machine running 5.8, 
and I'm trying to capture some informa¬ 
tion with ps. Something keeps going awry' 
with a job in the middle of the night, and I 
want to verify which arguments were passed 
to the command. I’ve been using ps -ef, but 
that seems to truncate the output. 



A /usr/bin/ps will pull up to 80 unmodi¬ 
fied characters straight from the kernel 
but stop there. The other caveat is that 
/usr/bin/ps separates the arguments by 
spaces, so if you had embedded spaces in the command line, it 
would be impossible to tell from looking at the ps output. 

If you’ve installed SUNWscpux (or SUNWscpu if you’re not on 
a 64-bit machine), you may want to try- /usr/ucb/ps. If you give it 
enough w flags (they’re cumulative), /usr/ucb/ps auwww will show 
all args, as modified by the process and stored in /proc. Programs 
like send mail modify their arguments, though, so this may not give 


A BIND processes the checks in order and then stops when it 
finds a match. If you want to make sure that both requirements 
are met, you can do so by changing your notslaves acl: 

include "keys.txt": 

acl slaves { 


So, based on this, zone transfers should hap¬ 
pen only if the request comes from 10.2.1.1 
and has the correct TS1G key. What I see is 
that 10.2.1.1 can transfer without a TSIG key. 
If 1 reverse the order to: 

allow-transfer { key tsigkey; ! notslaves; }; 

then anyone w'ith the key can initiate a trans¬ 
fer. How' do 1 AND them so that BOTH are 
required? 


you exactly what you’re looking for, either. 10.2.1.1; 

Solaris 9 contains a program called pargs in ); 

SUNWesu/SUNWesux that will examine a process or core tile and 

print the program's arguments and environment variables and values. acl notslaves ( ! slaves: any; } 


Q l’m running BIND 9.2.2. and I’m trying to lock down security 
on my zones via the named.conf configuration file. I want to 
ensure that transfers are only allowed from hosts that match a 
certain IP and that have a matching TSIG key. I can easily do one or 
the other, but I can’t seem to AND them together. Here’s a snippet 
of my config file: 

include "keys.txt"; 

acl slaves { 

10 . 2 . 1 . 1 ; 

}; 

acl notslaves { ! slaves; }; 
options { 

directory "/etc/named”; 

listen-on { 10.1.1.1; 1; 

allow-transfer { ! notslaves; key tsigkey; }; 

1 : 


Submit questions to: http://www.sysadminmag.coni/quest/ 


options { 

directory "/etc/named"; 

listen-on ( 10.1.1.1; ]; 

allow-transfer [ ! notslaves; key tsigkey; }; 


Q fm running sendmai! 8.12.9 on Solaris 9. I've been adding 
some anti-spam measures to my me file, but I’m having issues 
getting this one to work as expected. 

FEATURE(’dnsbl', 'proxies.blackho1es.easynet.nl', ‘"550 5.7.1 Open 
Proxy Serve*’ "$Mclient_name)" DENIED by easynet DNSBL see 
http://proxies.blackholes.easynet.nl/errors.htinl'", '')dnl 

The mail is rejected as expected, but I’m getting an error in my log file: 

Oct 23 17:03:12 mail hub.my.domain sendmai1[3102]: [ID 374821 mail.crit] \ 
h82IJEeM023344: SYSERR(root): rewrite: map clientjiame not found 
Oct 23 17:03:12 mailhub.my.domain sendmai1[3102]: [ID 374821 \ 
mail.notice] h82IJEeM023344: ruleset=check_relay, \ 
argl=cpe-066-056-232-106.ec.rr.com. arg2-66,56.232.106, \ 
relay=cpe-066 056-232-106.ec.rr.com [66.56.232.106], reject=550 \ 
5.7.1 Open Proxy Server)DENIED by easynet DNSBL see \ 

http://proxies.blackholes.easynet.nl/errors.html 
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I double-checked the documentation, and client_name should be a 
valid variable from which I can extract information. Why is send- 
mail refusing to give me the name/address of the offending host? 

A Your problem is that you used parentheses instead of braces 
when you specified client_name. You want “$&{client_name}” 
not “$&(client_name)”. 

Q l've just acquired a Sun Blade 100 that Em setting up as a 
headless box in the machine room. Sun’s default serial baud 
rate is 9600, but I know it can go up to 38400. I’ve tried changing 
this in /etc/ttydefs, but it doesn’t seem to work properly. I know it’s 
not the other end {a console server), because I can gel other 
machines to connect to the exact same port at the higher speed, and 
I can gel the Sun to connect to the port at 9600. 

A To begin. I recommend against setting the console speed to 
something non-standard unless you’re documenting it well. 
You don’t actually list what errors or incorrect behavior you experi¬ 
enced, hut there are multiple places where the console speed must 
be changed so that the serial port speed is consistent throughout all 
of the states the machine could be in. 

You said you’ve changed /etc/ttydefs, which should now have: 

console:38400 hupcl opost onlcr:38400:console 

instead of: 

conso 1 e:9600 hupcl opost onlcr:9600::console 

This modifies the port speed after the machine is done booting and 
init spawns ttymon. At any point before this, you're still going to be 
at the wrong speed. 

You must also change the OpenBoot ttya-mode setting so that 
the serial port speed is correct before the kernel loads (while you’re 
at the OBP, for example). From a running system: 

eepron ttya - mode=38400,8,n, 1 

You can also drop to the OBP and set it there with: 
setenv ttya - mode 38400,8,n.l,- 

Additionally. /kernel/drv/options.conf and /etc/ioctl.syscon contain 
output in stty -g format. The former specifies the port speed after the 
kernel is loaded, and the latter specifies it after init starts running. 


Notice To Our Subscribers 

Occasionally, SysAdmin makes its mailing list available to vendors of 
products we think our readers will find interesting. Current subscribers 
receive free information in the mail from these vendors. 

If you prefer that your name not 
be used in these mailings, please 
let us know. Just copy or clip this 
form and send it with your name 
and address to: 


hut before ttymon is spawned. The easiest wav to change these two 
files is to modify /etc/ttydefs and the eeprom data first and then 
reboot. Next, log in as root on the console, making sure that root has 
no stty information in any of its login resource tiles. Save the cur¬ 
rent stty information by running: 

/usr/bin/stty -g > /tmp/stty 

Create backup copies of /kernel/drv/options.conf and /etc/ioctl.syscon. 

In /kernel/drv/options.conf, replace the information between the 
quotes with the contents of /tmp/stty; 

ttymode s="2502:1805:bd:8a3b:3:1c:7f:15 :4: 0:0:0:11:13:la:19:1Z:f:17:16"; 

You can then just mv /tmp/stty over the old /etc/ioctl.syscon. To 
make sure that everything was modified correctly, power off the 
machine with init 5 and then watch the boot sequence after the 
machine is powered back on. 

Q l’m running FreeBSD 4.8-STABLE, and I want to build some¬ 
thing from the ports directory using different configuration 
options. I tried editing the makefiles in the sre directory, but they 
seem to get overwritten every time I try the build. I suppose 1 could 
just make the package from scratch, but I’d rather have it listed in 
the package database so it’s easy to update/remove/reinstall later. 

A Generally, the best way to modify the configuration options is 
to do so on the command line while you’re making the port in 
question. If you're making modifications from the command line 
(this assumes a Bourne-like shell), you can either override the exist¬ 
ing configuration options: 

CONFIGURE_ARGS="-optl=arg -opt2" make install 

or you can append to them: 

make CONFIGURE_ARGS="-optl=arg -opt2" 

You can also edit the port’s Makefile directly (e.g., 
/usr/ports/lang/perl5.8/Makefile) before you run make, but be aware 
that your modified Makefile will likely be overwritten by your next 
evsup. 

Q I have a couple of Nelra T1 105s running Solaris 8 that are both 
attached to the same external disk pack. I’m using SDS to set 
up a JBOD RAID on this disk pack. I want to set up a poor man’s 
failover and mount the disk pack on the second machine if the first 
one should fail. 1 know I could purchase software from Sun. but 1 
need to do this on the cheap. I’m just not sure where to start... 

A What you want to accomplish is possible, but it’s not supported by 
Sun. You want to create (using the met a set command) a diskset, 
an SDS object that can be shared exclusively and non-concurrently by 
two hosts. To get started, take a look at the SDS 4.2.1 User’s Guide: 

http://docs.sun.com/db/doc/806-3205. 

If you want a Sun-supported configuration, you need to use Sun 
Cluster as well. 
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Q I get a bunch of spam that has no To: address, so I thought I'd 
whip up a sendinail mleset to reject such mail. I thought it 
would be pretty trivial, since there already appears to be a mecha¬ 
nism to check the To: header, but it doesn't seem to be working. I'm 
running 8.12.9, and here’s the important part of my me: 

HTo: $>CheckTo 

SCheckTo 

R$@ $#error $@ 5.7.1 t: "554 ERROR: No user specified." 

The mail comes through with a blank To: header every time I try to 
test, so I’m guessing that I wrote my ruleset wrong. Could you point 
out where my mistake is? 

A There is no mistake in your rule per se, but there is a bug with 
sendmail. From the KNOWNBUGS tile in the sendmail source 

tree: 

Header checks are not called if header value is too long or empty. 

If the value of a header is longer than 12.50 (MAXNAME + MAX- 
ATOM - 6) characters or it contains a single word longer than 256 
(MAXNAME) characters then no header check is done even if one is 
configured for the header. 

Until that bug gets fixed, your best bet is to use a milter or, if you’d 
rather separate it completely from sendmail, procmail. The proc- 
mail rule to send messages with empty or space-tilled To: headers to 
/dev/null would be: 

:0 

* A To:( )*$ 

/dev/null 


Q l’ve been administering DNS for quite a few years now, but I 
recently came across a term I had never heard before, and I was 
wondering whether you could provide a definition/example. The 
term w'as in bailiwick, and it seemed to have something to do with 
name servers. 

A The reason you haven’t heard this term before is probably 
because the only place it’s generally used is in reference to 
djbdns or DJBs arguments against DNAME and A6. If you’ve 
mainly used BIND (or something else), then it’s unlikely you would 
ever come across this term. As far as I know, there is no formal def¬ 
inition for the term. My interpretation is that your bailiwick is the 
realm of things (a domain, for example) over which you have con¬ 
trol. The common usage of the phrase is along the lines of “keeping 
at least one name server for a domain in bailiwick.” meaning that 
the nameserver for my.domain should be contained within 
my.domain (e.g.. ns I .my.domain). 

There seem to be two main reasons for keeping a nameserver in 
bailiwick, speed and security. If the nameserver for my.domain is 
located in this.otherdomain, trust is spread out and more points of 
exposure are added to equation. Also, every out of bailiwick refer¬ 
ence creates an opportunity for a loop. In regard to speed, each out 
of bailiwick reference means more queries per lookup and more 
opportunities for delay. Also, caches must limit the number of 
queries and the amount of memory per lookup. Time or memory 
could be exceeded before that lookup is successfully completed. 


Q A user with a Sun Blade 1000 as his desktop wants to attach 
some USB devices. I know that not everything is supported by 
Sun. but is there was a canonical hardware compatibility list some¬ 
where? 

^^Sun maintains a list of tested/verified USB products at: 

http://www.sim.con/io_technologies/USB.html 

If you're running Solaris 9. the following documentation might also 
be useful: 

http://docs.sun.coir/db/doc/817-0798/6mgisnq8g7a-view 


Q I in running FreeBSD 4.7-STABLE as a PDC. I need to add 
users that end in $. but the adduser script keeps telling me that 
this is not a valid username. I’ve seen this done before, so I know 
it s possible, but I’m not sure whether there’s some additional argu¬ 
ment I need to pass to adduser or what. 

A You can add users that end in $ by using vipw instead of 
adduser, or you can modify /etc/adduser.conf to let adduser 
know that the username you're trying to enter is valid, 
/etc/adduser.conf contains a regular expression, usemameregexp, 
that usernames are checked against. The default is: 

usemameregexp = ,fl [a-20-9_][a-zO-9_-]*$’ 

To allow' users with dollar signs at the end. change this line to: 

usemameregexp = ,/ '[a-z0-9_][a-z0-9_-]*\$*$’ 


Q I have both gcc and the Sun compiler installed on my machine. 

I’ve been trying to build some software, but I have a feeling 
I m running into incompatibilities between the two compilers. My 
suspicion is that some of the libraries I ni trying to link with were 
compiled with one compiler and some were compiled with the 
other. 

A There are a few methods you can use to try to extract data. The 
most reliable will probably be: 

/usr/ccs/bin/mes *p /path/to/file 

/usr/bin/strings - /path/to/file |grep GCC (if it was compiled with 
GCC) 

The following may also be useful in helping debug your problem: 

/usr/ccs/bin/nm /path/to/file 
/usr/ccs/bin/dump -c /path/to/file 

Am y R,ch > president of the Boston-based Oceanwave Consulting , Inc. 
(http://www.oceanwave.com, has been a UNIX systems administrator for 
more than JO years. She received a BSCS at Worcester Polytechnic Institute, 
and can be reached at: pna@oceanwave.cont. 


December 2003 


www. s v s a c I n ? n mag .com 


Sys Admin — 41 



Vendor Sponsored Content 


PureMessage is 'Big Win' for 
Vignette Corp. 


Customer: Vignette Corporation 
Industry: High Tech 
Product: Sophos PureMessage 

Business Overview 

Vignette (Nasdaq: VIGN) is a global provider of content 
management and portal software. Headquartered in Austin, Texas, 
the company powers the Web applications of more than 1,600 
leading organizations. Vignette has over 850 employees in offices 
located throughout the Americas, Europe, Asia, and Australia. 

Business Challenge 

Vignette's spam problem appeared on the executive radar in 
early 2002, as users became overwhelmed with the increasing 
volume. The company estimates that the amount of spam was 
glowing by 5% each month, resulting in a flood ol helpdesk tickets 
and mounting employee dissatisfaction. 

The IT department also felt the impact, as they struggled to 
manage the inbox clutter and the impact on backend systems. Chris 
Brown, Unix Email Manager, likened the deluge to "trying to drink 
water out of a fire hose." 

Eager to solve the problem before spam levels increased 
further, Brown's team set out to define their requirements. After 
accuracy, top considerations included ease of management, 
comprehensive reporting, integration with existing infrastructure, 
and the ability to filter both inbound and outbound mail. 

Says Brown, "We wanted a product that had a clear plan of 
attack;' one that uses several methods to detect spam. It was also 
important that the solution didn't magnify support requirements, so 
software-based filtering was definitely our preference." 

The PureMessage Solution 

After evaluating six potential vendors. Vignette selected Sophos 
PureMessage. Explains Brown, "PureMessage had the lead from 
the start. It had the most complete feature set and lots of power, and 
the additional filtering tools were a bonus. 

"PureMessage was extremely effective during our testing 
phase: the capture rates were good, and the false positive rate was 
very low. If s a powerful solution, but still easy to use." 

The PureMessage feature set was a deciding factor, both for the 
comprehensive range of anti-spam tests, and additional filtering 
capabilities. Says Brown, "Other products we looked at lacked the 
ability to manage multiple services, whereas PureMessage gave us 
anti-spam, anti-virus, and policy enforcement, all in one package. 

"We're secure from external threats, and by filtering outbound 
mail we're also able to implement IP and confidentiality protection. 


Today, economic considerations are driving the desire to 
consolidate server functions, so single platform email management 
is a 'big win."’ 

The central, web-based console ensured that managing the 
software was trivial. Says Brown, "We can view and modify the 
rules—both local and the ones created by Sophos’s research team— 
without hassle. The interface is intuitive, allowing us to easily 
specify the features we want to use. From blackhole lists, to white 
and black lists, to heuristic rulesets, and bayesian word analysis—all 
these tools are at our fingertips." 

Results 

Even during the evaluation. Brown's team found that, "ROI was 
immediate. The amount of spam we were keeping out of our 
system was huge. When the executive team saw the figures, they 
pushed for PureMessage." 

Months later, the team remains certain they made the right 
decision. Says Brown, "Sophos has provided a great set of tools and 
rules. PureMessage is very good at recognizing advanced spam 
methods—the regular updates keep the capture rate extremely high. 
We have over 95% spam detection accuracy. 

"PureMessage has reduced email administration for my 
department. There are no more spam-related helpdesk tickets. And 
our users are very impressed with the effectiveness. It’s taken the 
spam headache out of their lives. 

"It’s apparent to everyone that PureMessage is an awesome 

product." 

Contact Sophos 

Toll-free in North America: 1.866.866.2802 

Email: pmsales@sophos.com 


For a free, fully supported evaluation, see: 

www.ActiveState.com/PureMessage/more 
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TOOLS 


Cross-Platform Native Package 
Creation with EPM 

Jeff Layton 


I 'm a big fan of open source software and also of writing my 
own scripts and programs to automate things. Currently, I man¬ 
age three major Unix-ish operating systems at work: Solaris. 
HP-UX, and Debian GNU/Linux. Tracking all of this added soft¬ 
ware can be a relatively large burden, especially if it’s just copied 
or unzipped into place. Because of this. I've recently come to favor 
using the “native" package format for an operating system. If you 
simply copy or untar files on a machine, or use a non-standard 
packaging scheme, it's quite difficult to know when you are over¬ 
writing a file that is being tracked in the native package catalog, 
and to uninstall the files later. 

Unfortunately, the process to generate a package for each OS 
often varies greatly between platforms. Generating HP-UX depots 
is very different from creating a Solaris package. Even on Linux, 
there are significant differences between the two major package for¬ 
mats (RPMs and debs). 


The EPM distribution is available from: 
http ://www. ea sysw.com/epm 

EPM is licensed for distribution under the GNU Public License. To 
begin, download the tar ball and unpack it into a directory. It requires 
an ANSI C compiler, a make program, a Bourne-style shell, and 
gzip. If you want to build in support for the GUI setup program, 
you'll also need the FLTK library (http: //www. fl tk.org). 

Building and Installing EPM 

EPM is an autoconf-managed project, so look over the 
INSTALL and README files, and then issue the command: 

% ./configure 


The Easy Software Productions Package 
Manager (EPM) 

Enter EPM, the cross-platform packaging tool from Easy 
Software Productions (better known for their fabulous CUPS soft¬ 
ware). EPM can generate binary packages for many Unix and 
Linux-based operating systems, including AJX, BSD. Tru64, 
Debian, HP-UX. IRIX, OS-X, Red 
Hat, and Solaris. It can also build 
packages in its own format that 
includes a GUI setup program. In 
this article, 1 will focus on creating 
"native" packages —- those that use 
the packaging scheme bundled with 
the OS. 

EPM relies on programs that are 
bundled with an operating system to 
package software. Building an RPM 
package means that you need an 
"rpm" binary, and building a pack¬ 
age for Solaris requires the Solaris 
"pkgmk” tool. Building packages in 
non-native formats is possible, but 
only if you have a version of the 
packaging tool available on the sys¬ 
tem on which you're building the 

package. For instance, it's possible to build an RPM package on a 
Solaris machine, but an installed version ot RPM must exist on your 
Solaris machine. 





Or, do the following if you want to install it in a different location 
from /usr/local: 

% ./configure - -prefix=<destination> 

Now’, build the software with: 

% make 


Once the software is built, how¬ 
ever. do not do the usual “make 
install". Instead, you can make 
EPM a package in the native format 
of the machine with the following 
command: 

£ ./epm -f native epm 

On some architectures (notably. 
HP-UX). building packages 
requires root access, so you may 
need to be root to execute this 
command. The “native" keyword 
here tells EPM that we want to 
build a package in the native for¬ 
mat for the machine on which 
we re working. It's possible to 
different package formats (e.g.. “deb" or "rpm” instead of 
as long as the proper tools for packaging the software 
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exist on the system. See the EPM documentation for a list of key¬ 
words for different package types. 

After the command completes, the generated package should be 
inside a directory named after the OS and architecture of the 
machine (e.g., solaris-2.8-spare or linux-2.4-intel). Install this pack¬ 
age using the native package installation tools for your platform 
(e.g.. dpkg on Deb i an. or p kg add on Solaris). 

Overview of the Packaging Process 

The first thing to do when building a package is to obtain the 
sources and produce binaries. EPM doesn’t compile the software 
for you; it only packages the files for distribution. 

Once you generate all the files you want packaged for distribu¬ 
tion, you'll need to write a configuration file that describes your 
package. These files are called “listfiles” and describe meta-infor¬ 
mation on the package itself, which files get installed, what their 
permissions and ownership are, and whether there are scripts that 
should be executed when the package is installed or removed. 

Building a Package 

For this example, we will package a copy of OpenSSH (avail¬ 
able at http://openssh.com), built for installation in /usr/local, 
with configuration files in /etc/ssh. 

Download the sources for the latest portable distribution and 
unpack them into a directory. Presuming you have all the necessary 
dependencies, run: 

./configure --prefix-/usr/local --sysconfdir=/etc/ssh 
make (or gmake) 


Again, we skip the “make install” stage, because we’ll be putting 
the files into a package for distribution. 

Creating ListFiles 

EPM uses package description files called ''listfiles''. These files 
are named with the name of the package and end with the extension 
".list". For example, "openssh.lisf' would be the listfile describing 
the “openssh” package. Listfiles are documented in epm. 1 ist(5). 

Package Identification and Information 

Every listfile must contain a set of required information, Here's 
an example of the required info for “openssh.list”: 

^product OpenSSH 

^copyright The OpenSSH Team, All Rights Reserved 

^vendor http://www.openssh.com/ 

license ./LICENSE 

^readme ./README 

^description Secure Shell 

Aversion 3.6.1p2 3612 

The ^product, ^copyright, % vendor, and ^description are sim¬ 
ple text fields that will populate similar fields in the generated 
package, ^license and %readme are names of files that contain 
their respective information. The %version line has two fields. 
The first is a human-readable text version. The second is an inte¬ 
ger version number. If the integer version number is omitted. EPM 
will try to generate one from the human-readable field. 

Files 

Next, we can start adding files and 
directories to be packaged. The basic for¬ 
mat for filesystem entities is: 

type mode owner group destination source \ 
options 

We’ll build this package with the build 
directory of OpenSSH as our current work¬ 
ing directory. So. to add entries for the SSH 
program and its manpage, we'll add these 
lines to our listfile: 

f 755 root sys /usr/local/bin/ssh ./ssh 
f 644 root sys /usr/local/man/manl/ssh.l \ 
ssh.l.out nostripO 

By default. EPM runs the Strip(l) com¬ 
mand on all files that it packages. The 
strip command removes symbols from 
object files, thereby reducing the size of 
binary' programs at the expense of some¬ 
times useful debugging information. To 
prevent this behavior, add the nostripO 
option. Since manpages are not binaries, 
stripping has no effect; but on some plat¬ 
forms. it generates a warning. 

Directories 

You can also add directories to the pack¬ 
age. Most packaging systems will create 
directories for you if you install a file into a 
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directory that doesn’t exist. However, if you want to create empty 
directories, have control over permissions and ownership, or want 
the directories removed on package removal, then you must specify 
directory entries. Here we‘11 create the directory that will hold the 
OpenSSH configuration files: 

d 755 root sys /etc/ssh - 

The “source” attribute for directory entries is always a hyphen, 
since they aren’t actually copied from anything. 

Symbolic Links 

Symbolic links are prefixed with “1” in the listfile. Here we add 
the slogin link, which points to the SSH binary: 

1 111 root root /usr/local/bin/slogin ssh 

The source field for symlinks is interpreted as is, so you can create 
relative or absolute sym links. 

Configuration Files 

Some packaging systems (e.g,, Debian and Red Hat) handle con¬ 
figuration files differently from regular files. Others (such as HP-UX 
and Solaris) make no such distinction. If your packaging system 
does distinguish between configuration and regular files, then you 
can declare configuration files like this: 

c 644 root root /etc/ssh/ssh_corfig ssh_config 


For packaging systems that make no distinction between the two, 
configuration files are treated as normal files. 

With the above line in your listfile, what happens if this tile 
already exists at install time? It depends on the packaging system 
used. With Debian, for example, you are presented with a dialog 
asking whether you want to overwrite the file. On Solaris and HP-UX, 
this file is treated like any other and is overwritten. 

If in doubt, create a test package for your platform and see how 
it handles overwriting an existing tile before relying on this feature. 

Variables 

Arbitrary variables are defined in listfiles by prefixing them with 
a dollar sign. Since we’ll be installing a lot of our files under 
/usr/local, we can set a variable to that location and reference it 
throughout the listfile: 

$prefix=/usr/local 

We can then reference this as either Sprefix or $ (prefix). If you 
do not use curly braces to reference them, then variables names end 
at the first slash, hyphen, or whitespace. 

If we use the $pref i x variable above to describe the installation 
prefix, our file and symlink entries in the listfile might look like 
this: 

f 755 root sys $ {prefix} /bin/ssh ./ssh 
f 644 root sys ${prefix]/fnan/manl/ssh.l ssh.1.out nostrlpO 
1 777 root root $tprefix}/bin/slogin ssh 
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By using variables to describe installation locations, it becomes 
easy to change the location of package files. Now, if we want to 
change the package so that it installs under /usr instead of /usr/local. 
all we need to do is change the $prefi X variable. 

Variables also make it easy to build list tiles programatically. and 
when combined with conditionals (described later in this article) 
make it possible to describe very complex packaging scenarios. 

A Simple Example 

Listing 1 is an example listfile, describing a basic SSH client 
package. (Listings for this article are available from the Sys Admin 
Web site at: http://www.sysadmimnag.com.) After building SSH 
and editing the ssh_config Hie to your liking, copy this file into the 
OpenSSH source directory and name it ssh-clSent.list. Then run: 

epm -f native ssh-client 

This should build the package and place it into a directory named with 
a combination of your OS and architecture (e.g., "lintix-2.4-inter or 
“solaris-2.8-sparc”). You can then install this package using the 
proper package installation tools. 

Scripts 

Most packaging systems provide hooks to allow you to run arbi¬ 
trary code during the installation or removal process. EPM supports 
this via the following header tags: 


^preremove 

%postremove 

^preinstall 

/Spostinstall 

EPM can either he told to run a particular command, or scripts can 
be embedded in the package. For example, this line in a listfile will 
have EPM generate an SSH host key after installation: 

%postinsta11 /usr/local/bin/ssh-keygen -q -b 1024 -t dsa *U ” 

Only one script tag of each type is allowed, but you can embed 
scripts that are in other tiles with a less-than sign: 

Spostiinstall <tIsredir]/generate_keys 

Alternatively, you can inline scripts using here file-like syntax: 

%postinsta11 <<E0T 
printf "Generating keys: " 

/usr/1ocal/bin/ssb-keygen -q -b 1024 -t dsa -N " -f \ 
etc/ssh/ssh_host_ds a_key 
printf "done'' 

EOT 
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Package Dependencies 

One great thing about using packaging systems is that it facili¬ 
tates tracking dependencies and incompatibilities. EPM allows you 
it) declare a package requirement using the % requires tag. For 
example, if we dynamically link OpenSSH. we should require the 
OpenSSL package to be installed to ensure that it will work. We'll 
also declare that we need version 0.9.7: 

^requires openssl 0.9.7 

We may obtain packages from other places that depend on the same 
software that is in our package, but under a different name. The 
%provides tag allows us to ‘"alias” our package: 

^provides ssh 

We can also declare an incompatibility. Here we’ll declare that this 
is incompatible with the kerberized version of SSH: 

ncompat sshkrb5 

This states that this package can replace a packaged commercial 
version of SSH that is named “ssh-commercial”: 

/{replaces ssh-commercial 

Conditionals 

It’s possible to have sections of your listfile that are applica¬ 
ble only on certain packaging systems, operating systems, or 
architectures. The %format tag declares that anything following 
it is applicable only for the declared packaging system types. 
The special name “all” refers to all format types. In this exam¬ 
ple. anything following this line applies only to RPM. deb. and 
Solaris packages: 

^format rpm deb pkg 

The ^system tag is similar to the %format tag, but it works on the 
system name as reported by a lowercased linarne -$ and an optional 
OS version as given by uname -r. In this example, we change the 
package dependency on Solaris to require the OpenSSL package 
provided by sunfreeware.com: everything else will depend on a 
package called “openssl”: 

/{system Solaris 
/{requires SHCossl 
^system Isolaris 
^requires openssl 

There are also a number of conditional tags that can operate on 
variables: 

%if 

Slfdef 
%elseif 
iSel seifdef 
Seise 
%end i f 

These are primarily of use when you have listtiles that are generated 
through some automatic process. 
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Init Scripts 

EPM treats System V init scripts a little 
differently. Init scripts are declared much 
like normal files, but are prefixed with an 
“i" instead: 

i node user group service-name source \ 
["options"] 

The service-name in this case is what the 
script that is installed in the normal init 
script directory should be named. This 
varies by platform, but on Linux and 
Solaris, it usually gets installed in 
/etc/init.d, and on HP-UX in /sbin/init.d. 

The runl evel () option lets you control 
what rc directories should get symlinks that 
point to the init script. Any non-zero run- 
level listed will get a start symlink. whereas 
run level 0 will get a stop symlink. 

The start!) and stop!) options allow 
you to specify at what point in the startup 
and shutdown process the script should be 
run. Here's an example: 

i 755 root root sshd sshd.rc "runlevel(02) \ 
start{ 82 } stop( 18 )” 

This would install the sshd.rc file as 
"sshd" in the init script directory. It would 
then create a symlink in the rcO.d direc¬ 
tory called KIBsshd, and one in the rc2.d 
directory called S82sshd. There are also 
some Apple OS X specific options that 
handle its dependency-oriented init proce¬ 
dure. See the documentation for more 
info on it. 

A Complete Example 
Package 

Listing 2 shows a listfile for a complete 
OpenSSH package, which uses most of the 
features of LPM Fve covered here. As 
before, copy this file into the directory 
where you’ve built SSH and name it 
“openssh.list". Edit the configuration files 
to your liking and add any init scripts that 
are not part of the distribution. Then run: 

epm -f native openssh 

It should generate your package and place 
it into the appropriate directory. 

Generating listfile Entries 
Automatically 

Manually configuring list files for a 
large package can be very tedious, so 
EPM comes with another command called 
fnkepml i St. which will automatically gen¬ 
erate listfile entries for everything under a 
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given directory. When generating a pack¬ 
age with a lot of files. 1 often do an initial 
install in a temporary directory and use 
mkepml i st to generate the list of files that 
would be installed. I then do a search and 
replace to change the locations of the files 
to their actual location. This is particu¬ 
larly useful when installing a lot of files 
in a directory shared among many pack¬ 
ages (such as /usr or /usr/local). 

Conclusion and Resources 

EPM has transformed how we 
approach software maintenance at my 
site. By making it simple to generate 
custom packages, ue can install software 
in a consistent manner across many 
machines as well as track versions and 
dependencies. This makes it much easier 
to deploy “standard" machine configura¬ 
tions and ensure consistency across 
many machines. 

The EPM site is the primary site for 
EPM-related news and information 
(http: //www.easysw.com/epm). It also con¬ 
tains more comprehensive documentation 
on using EPM. 

Major kudos and thanks to the ESP 
folks for making another great software 
tool, and for being so helpful via their 
newsgroups and mailing lists. 

Jeff Layton has been working with computers 
since he got his paws on a TRS-80 in 1981 at age 
10. After working in the Macintosh and PC realm 
for more than a decade and attaining a degree in 
Physics Education, he came to his senses and 
made the jump to Unix administration in 1996. He 
is currently employed as a Senior Unix Systems 
Administrator at Bowe Bell and Howell. 
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CONFIGURATION 


Managing Network IP Assignment 
with a Database and Web Interface 

Jim McBride 


T he Gar van Institute of Medical Research is a growing organi¬ 
zation with about 350 researchers and support staff, and 
roughly the same number of computers throughout the build¬ 
ing. We recently ran out of IP addresses within the two subnets we 
operated, so we launched a project to remodel the IP address space 
and reconfigure each computer with a new IP address. This required 
a visit to each computer in the organization — an arduous task that 
we prefer to avoid. 

Our IT group is the appropriate size for a publicly funded med¬ 
ical research group — meaning that we have the bare minimum 
number people to run the operation. 

Reconfiguring each computer w ith a 
new IP address would add up to 
about 30-40 hours work. Since we 
had been assigning IP addresses 
directly to computers as they 
arrived, there was no way to avoid 
manually configuring each com¬ 
puter. Asking the researchers to 
adjust their own IP control panels 
was out of the question. Thus, w’e 
wanted to make this the Iasi time w'e 
would have to take this manual 
approach. 

The practice of statically assign¬ 
ing an IP address to each computer 
in the network also had drawbacks 
we w'anted to remove. The most sig¬ 
nificant problem of statically 
assigned IP addresses is the need to 
keep an accurate database (or even just a simple list) of IP addresses 
in use and tree IP addresses available to assign new computers. We 
found that the list of IP addresses would become out-of-date over 
time and we would assign a pre-existing IP to a newly arriving com¬ 
puter. This usually happened when the IP address was assigned to 
an offsite laptop computer, so the usual ping test to see whether the 
IP address was in use provided a false-negative response. 

'fhe final goal of the project was to increase network secu¬ 
rity. We were about to upgrade an aging set of 10-MB hubs for 
100-MB Cisco switches. The Cisco gear allowed us to assign 
computers into VLANs and define access rules in and out of 
each VLAN. In this way, we could make the majority of our 
user’s computers invisible to the Internet and therefore more 
secure from network attack, leaving only the public server 
equipment exposed to the Internet. We planned six VLANs with 
specific access rules for the types of computers they contain. 
For us, a VLAN is defined to be a set of contiguous IP 
addresses. So, again, good management of the IP address space. 


and the association of specific IP addresses to specific computers, 
became increasingly important. 

Our Solution 

We w'anted the centralized address management offered by 
DHCP, bul for security reasons, we w-anted to associate a specific 
computer with a specific IP address. Our solution was to assign a 
fixed IP address to each computer through DHCP. The computer 
would retain the fixed address, but it would discover that address 
each time it rebooted and requested a session IP from the DHCP 

server. Wc assigned an IP address to 
the computer by associating the IP 
address with the computer's MAC 
address in the DHCPD.CONF file. 
The DHCP.CONF file therefore 
contained an up-to-date fist of 
device to IP address mappings. 

If we wanted to alter an address, 
we could change the IP address 
assigned to the computer in 
DHCPD.CONF. restart DHCP, and 
tell the user to reboot. After the ini¬ 
tial setup. w ; e could manage the IP 
configuration without ever having 
to visit the client, bul the system 
still provided the equivalent of static 
address assignment by maintaining 
a persistent association of an IP 
address with a specific device. 

When 1 thought about the 
process of visiting each computer and resetting it for DHCP, it 
involved several steps: 

1. Record the computer MAC address. 

2. Record the computer details (operating system, disk size, network 
patch point. VLAN membership, etc.). 

3. Update control panels to use DHCP for IP. 

4. Allocate an appropriate IP address from a fist of free IP 
addresses. 

5. Edit DHCPD.CONF with the MAC address and the newly allocated 
IP address. 

6. Restart DHCP. 

7. Reboot the computer. 

8. Confirm die computer obtains the correct IP address and is 
operational. 

Given that all computers on the network needed to be cut over in a 
short period of time, we wanted to avoid manual edits of 



www.sysadminmag.com 


50 — Sys Admin 


December 2003 








Setting the standard for 



INSTITUTE 

Conference Schedule 
November 2003 - 
April 2004 


SANS Network Security 2003 

New Orleans, LA Nov. 14-19, 2003 

SANS Phoenix Retreat 

Phoenix, AZ Dec. 1-6, 2003 

SANS NIAL VI 

Washington, D.C. Dec. 4-5, 2003 

SANS Cyber Defense Initiative 

Washington, DC Dec. 8-13, 2003 

SANS Biscayne Bay 2004 

Miami, FL Jan. 12-17, 2004 

SANS Darling Harbour 2004 

Sydney, Australia Jan. 12-17, 2004 

SANS CDI West 2004 

San Diego, CA Jan. 26-31, 2004 

SANS NIAL VII 

San Diego, CA Feb. 2, 2004 

SANS Peachtree 2004 

Atlanta, GA Feb. 9-14, 2004 

SANS Leadership Kauai 2004 

Kauai, HI Feb. 10-13, 2004 

SANS Tysons Corner 2004 

Tyson's Corner, VA Feb. 23-28, 2004 

SANS North Pacific 2004 

Portland, OR March 3-8, 2004 

SANS Silicon Valley 2004 

San Jose, CA March 15-20, 2004 

SANS Aloha 2004 

Honolulu, HI March 21-26, 2004 

SANS 2004 

Orlando, FL April 2-7, 2004 

Walt Disney World Dolphin 

SANS Online Training 
Instructor Led Online Training 
Local Mentor/Instructor Program 
Onsite Training 
Awareness Training 

GIAC Prep Teaching Kits and 
Practice Tests 


SECURITY TRAINING 


SANS 
Security 
Essentials 
and the 10 
Domains 


Firewalls, 

Perimeter 
Protection, 
and VPNs 


Intrusion 

Detection 

In-Depth 

Track 3 


! BBS 


Hacker 
Techniques, 
Exploits, & 
Incident 
Handling 

Track 4 

-j? 




an 


special event for Managers and 

Ml 


Leaders 


-Jf 




■M 


A special event for Managers and 


Leaders 








■M 


•M 

~jr 

-JJ 


Ml 

jj 




■M 






rity Awareness 






Auditing 
Networks, 
Perimeters 
& Systems 

Track 7 

-jr 

Ml 


System 
Forensics, 
Investigation, 
& Response 




Ml 


4 R 






Program 


■jj 






SANS 
17/99 
Security 
and Audit 
Framework 




Ml 


SANS 

Security 

Leadership 

far 

Managers 

Track 12 


Ml 


Ml 


Hands-on Training 
V Vendor Exposition 


Ml 


-J7 


Ml 




Ml 


Information subject to change. For the most up to date information please see www.sans.org 









































Figure 1 Screen shot of an asset in browser 



, http //ttakag* g^a»vun*w,EDu.AU,L„[$/edlLph& 




fess! 




O 


.....j w hHr 71 Hhtogt ijrjw Et>U. WJ A? s*U /Mit pfi& ? *a i*i &*$?? 


4ffl 


Assets WEB Version 1.1 


jimmcb 


AsM.iL.abv3: J 
Computer: [Mamum 
U* allocation \ Select a VLAN t 
LPNci: 


jiteYSt"' 


Mat’A tki rtf &: pHI;»3C if ■ i?; 54 " 
CIS: joix 

DHCP On f^T™ 


Room: | 

Chtmcn |jim McStaie Comment: P™* 

Program D St S - in ftjunift urn 1 echno lo gy New i firogram; j No Change 

IPSubNet: poF _ “ IPNode: JSS5~ 

HOKiN amt: |jimG4 DN S 

Meuwry: pr£ ! Disk: |oooG3 

Patch: I 


[ fmtaflgd Software [ 

^ ravfr 


Figure 2 

Screen shot of main menu, showing DHCP restart and software link 
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DHCPD.CONF and hand-written IP address 
lists. We planned to use the Web and a 
browser interface to manage a database of 
computer details. We already operated 
several Sybase databases for research 
applications, and a number already had 
PHP pages written allowing information 
management via a browser. We set up a 
table in Sybase with the attributes shown 
in Table 1. 

We presented this Sybase table via PHP 
code to a browser {see Figure 1). Any 
member of the IT Group can authenticate 
and access this system. The PHP code 
allows the authenticated IT staff member to 
add and edit records in the Sybase 
Computers table. The IT staff member fills 
in all the fields from AssetLabel. Room, 
Computer, and so on through to DHCPOn 
and Patch, with the exception of IPNet, 
IPSubNet. and IPNode. The PHP code cap¬ 
tures all these entered attributes and, using 
the VLAN selected, executes the SQL code 
to determine the next available free IP 
address for that VLAN (see Listing l. ipas- 
sign.php). The fields IPNet. IPSubNet, and 
IPNode are set and the record saved to the 
database. That IP number is no longer free, 
but is associated with the MAC address of 
the computer being connected to the net¬ 
work. The IP address cannot be reassigned 
until the Sybase record is later edited and 
the IP address is deleted or changed. The 
IT Group no longer must consult lists of 
allocated and free IP addresss. The PHP 
code looks after it all. 

If the computer leaves the network, such 
as when an employee leaves and takes his 
personal laptop w ith him, the IT Staff mem¬ 
ber simply logs into the Web page, locates 
the computer, and removes the assigned IP 
address from the database record, If the 
computer moves from one VLAN to 
another, the IT Staff member logs into the 
Web page, locates the computer, and selects 


Table 1 Attributes of the computer database table 

AssetID 

Unique identifier for the computer or printer 

DNS : If the computer is registered with a name in DNS 

Room 

Location of the device 

MacAddress Device mac address 

TeamID 

Name of research group operating the device 

HostName j Hostname for DHCP 

Computer 

Type of device 

ComputerComment Any comments 

AssetLabel 

Written asset label on the device 

OS Operating system 

Owner 

Primary operator of the device 

ComputerMemory Amount of installed memory 

IPNet 

IP net 

ComputerDisk Size of local disk 

IPSubNet 

IP subnet 

DHCPOn If the device is under DHCP control 

IPNode 

IP node number 

Dateln Date this record was last changed 

Patch 

1 Cable patch number 

DateOut Date this record was replaced (used for tracking 

: billing history) 
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Listing 1 ipassign.php 


<? 

/‘START IP ALLOCATION SECTION*/ 
foreach (SRanges as $ Key => IValue) f 
if (ilPMode — iKey)[ 

$0efinedRange - TRUE: 
break; 


if C$DefinedRange = TRUE){ 

//The option is in the ranges array 

IThisftange = SRanqes[£IPMode]: 

if ($TPNet!="" or iIPSubNet1="" or $IPHode!“””> I 

echo XBRXBRXfont color=\"#FF0000\”Xb>Error: Can't select $ThisRange->Label mode \ 
of ip allocation and specify manual IP. Record not added</bX/font><BRXBR>"; 
exi t; 
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the new VLAN from the dropdown list. The 
PHP code again determines the next free IP 
number in that VLAN range and assigns it to 
the database record. The old IP immediately 
becomes free and can be assigned by the 
system at a later date, 

V 

]. Shows up at a computer 

2. Determines the MAC address and other 
relevant information 

3. Goes to the configuration page through 
the browser interface 

4. Enters the data. (Does the PHP script 
generate some of the settings?) Clarify, 

5. Selects a VLAN for the computer from a 
dropdown list 

6. The PHP code completes the table by 
assigning an unused IP address associ¬ 
ated with the VLAN. 

If the computer is moved or joins a differ¬ 
ent VLAN, the IT staff member modifies 
the configuration through the browser. 

Since DHCP is running from the same 
server that hosts Sybase. Apache, and 
PHP. it became u simple task to add PHP 
code to generate the DHCPD.CONF file 
from the Sybase table and restart DHCP 
(see Figure 2). The Sybase table had 
become the source of the DHCPD.CONF 
File. If there are any errors in the Sybase 
table, they become immediately apparent 
when DHCP restarts and the computer is 
restarted and does not get the correct IP 
address. This prevents discrepancies 
between the Sybase records and the real 
world. 

What We Did 

We set up the table in Sybase, wrote the 
PHP code to manage the table and to recre¬ 
ate DHCPD.CONF and restart DHCP. The 
IT Group then scattered through the build¬ 
ing. visiting each computer and performing 
the steps outlined above. The project ran 
smoothly to completion. We migrated 
about 350 computers from static IP address 
allocation over to DHCP control, generat¬ 
ing a current register of all computer and 
printer hardware on the network at the 
same time. 

Now when a new computer is to be 
commissioned, or a computer is relocated 
or moved between VLANs, any member of 
the IT Group can access this system, update 
the Sybase record, and implement the 
changes into DHCP — all from a browser. 
We often use the browser running on the 
computer that is being moved or updated. 
We also use browsers on computers out in 
the research laboratory areas rather than 
having to run back to our offices. 
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Listing 1 continued 


} 

if ($ThisRangeASubSize — 1){ 

// Only one subnet, get the next free IF for this range 
$S>baseQjery“"SELECT min(IP) FROM IgSybaseDB..FreelP WHERE IP not in 
(SELECT IPNode from IgSybaseDB. .IgSybaseTable where IPSubNet—". $ThisRange->Subn-e'ts[0]." \ 
and DateQut-nulI) and IP >- tlhisRange->NodeLower and IP <= $ThisRange->NodeUpper"; 
ISybaseResult=syDase_query(JSybaseQuery,SSybaseLink): 

$MyRow=sybase_fetch_row($SybaseResu11); 

1 i st(la, $IP)=-each( IMyRow); 
if (IIP-""} ! 

echo "<BRXBRXfont color=\"#FF0000V’Xb>Severe error; no more free ips in the \ 
specified range. Abort</bX/font><BRXBR>"; 
exit; 

1 

tIPSN - IThisRange->Sufcnets[0]; 

J 

else { //Range spans multiple subnets 
//look for ips in the first subnet 

$SybaseQuery="SELECT min(IP)FROM IgSybaseDB..FreelP WHERE IP not in 
(SELECT IPNode from IgSybaseDB. .IgSybaseTable where IPSubNet-".IThisRange->Subnets[Q]." \ 
and DateOut=null) and IP >= IThisRange->NodeLower"; 

ISybaseResult=sybase_query(ISybaseQuery,$SybaseLink); 

IMyRow=sybase_fetch row(ISybaseResult); 

1ist($a,IIP)=each(IMyRow); 
if (IIP—""} { 

//No ips in the first subnet, cycle through the rest 
for ($i=1; Si < IThisRange-OSubSize * 1; Ii++){ 

$ Syba seQ uery="S E LECT min(IP)FROM IgSybaseDB..FreelP WHERE IP rot in 
(SELECT IPNode from IgSybaseDB,.IgSybaseTable where \ 

I PSubNet-". $TM sRange • >Subnets [ $i ]." and DateQut-null)”: 

ISybaseResult=sybase_query(ISybaseQuery.ISybaset ink); 

IMyRow=sybase_fetch_row(ISybaseResul t); 
listda ,IIP)=each( IMyRow); 
if ((IP!-"") { 

//Valid IP has been retrieved 
IIPSN = SThisRange*>Subnets[Ii]; 
break; 

} 

} 

if (IIP--"") [ 

//IP still not allocated, try last subnet 

$SybaseQuery="SELECT min<IP)FR0M IgSybaseDB..FreelP WHERE IP not in 
(SELECT IPNode from IgSybaseDB..IgSybaseTable where \ 

I PSubNet-”. STfii sRange->Subnets [$Thi sRange->SubSi ze-1]." and DateOut-null) and 
IP <= $ThisRange->NodeUpper"; 

ISybaseResult-sybase_query(ISybaseQuery,I Sybase Link); 

$MyRow=syba se_fetc h_row(ISybaseResult); 

1ist(la,IlP)-each(IMyRow); 
if (IIP--"") { 

//No IP able to be allocated, error. 

echo "<BRXBRXfont color-\"#FF0000\"Xb>Severe error; no more free ips in \ 
the specified range. Abort</bX/fontXlRX6R>"; 
exit; 

} 

IIPSN - $ThisRange->Subnets[$ThisRange->SubSize-l]; 

) 

} 

else { 

IIPSN = $ThfsRange->Subnets[0]; 

) 

) 

IIPNet - IgDefaultIPNet; 

} 

el seif (IIPMode—"Manual") { 

// Manual IP allocation 

echo "<BRXBR>IPMode Manual<BRXBR>": 

if (IIPNet—"" or $ I PSubNet—”" or I IPNode—"”) ( 

// Allow blank IP. may be an non-ip connected device 

} 

else i 

// --- Does this IP address already exist? Duplicates allowed, only warn 

ISybaseQuery-’SELECT IPNode FROM IgSybaseDB. .IgSybaseTable WHERE IPNet=\"IIPNet\" and \ 

IPSubNet-JIPSubNet and IPNode-IIPNode and DateOut-null": 

ISybaseResult = sybase_query(ISybaseQuery, ISybaseLink); 

IMyRow=sybase_fetch_row(I SybaseResult); 

1 ist(Ia.Ioount)-each(IMyRow); 
if ((count!-’") ( 

echo "<BRXBR>Warning; The IP address IIPNet.IIPSubNet.IIPNode is already in use. \ 
The record <b>HAS</b> beer assigned<BRXBR>"; 

} 

] 

11PSN—11PSubNet; 

$ IP—$IPNode; 

} 

else ( 

//The allocation is neither manual or in the range array i.e. error. 

echo "<BRXBR>Unknown IP allocation mode IIPMode. Record not added<BRXBR>”; 

exit: 

I 

echo "<BRXBR>Allocating IIPNet.IIPSN.JIP<BRXBR>"; 

/*END IP ALLOCATION SECTION*/ 

?> 


Other Benefits 

The major benefits of this system are 
the management of the TP address space, 
the up-to-date record we now have of all 
computers and printers connected to the 
network, and we no longer must visit each 
computer on the network. 

We’ve experienced other benefits from 
introducing an improved information 
management system. We are part of the 
Australian Academic Research Network 
(AARNET). providing our Internet con¬ 
nectivity through our association with the 
good people at the University of New 
South Wales. We are hilled quarterly in 
one lump sum for usage based on the 
number of connected devices, and the 
traffic generated by each device. As an 
Institute, we agreed that these charges 
should be passed on to the various 
research groups. I needed to take the 
quarterly invoice and assign costs to each 
research program based on their actual 
contribution to the total invoice amount. I 
was able to obtain an online copy of the 
invoice, which includes a line for each IP 
address w ith its cost of traffic. I created a 
Sybase table to store the details of the 
quarterly bill. I wrote a Sybase stored pro¬ 
cedure to take each research group, iden¬ 
tify all computers operated by the group, 
sum the cost for the IP address of each 
computer, and produce a report. This 
report effectively becomes an invoice that 
we use to recover the costs from the 
research groups. 

We also now have the ability to track 
software loaded on computers. We added 
several tables to the database to record pur¬ 
chased software, including version and 
number of licenses purchased. We added a 
page to the Web interface to allow IT staff 
to update these details as software is 
installed. In this way, we also have a 
method of tracking what software is 
installed, where, and how many licenses 
are in use. 

Jim McBride gained a BSc from Sydney 
University then worked in the petroleum and 
insurance industries before moving to medical 
research. For the pas! seven years, Jim and the 
rest of the IT team have managed the computer 
and communication systems for the Garvan 
Institute o f Medical Research. As well as deliver¬ 
ing good network services, Jim and the team have 
designed an information architecture that now 
underlies in-house applications that scientists at 
the Garvan Institute use to meet the challenges of 
medical research. 
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CONFIGURATION 


HTE: Host Table Editor 

Daniel Speers 


M any years ago, 1 worked for a large trading firm in New 
York with about 100 different networks spanning 12 
offices around the United States and a few in other coun¬ 
tries. The network subnets were grouped by DNS domains and man¬ 
aged by several different systems administrators. IP addresses were 
managed in a central host table and in DNS zone files, all on one 
master server. This created several problems, not the least of which 
was keeping the /etc/hosts file and the DNS zone files in sync. 

Other problems involved the security of different people editing 
these tables, keeping track of what addresses were really in use. and 
what devices were using those 
addresses. We also had reserved 
addresses. This typically meant that 
the first 20 addresses on a subnet 
were for network devices, and the 
following 10 were for DHCP- 
assigned addresses. 

The idea to manage the host 
table in a database evolved from 
these problems. From this database, 
we could generate a proper 
/etc/hosts file, DNS configuration 
and zone files, and the DHCP con fig 
files. Address space could be 
reserved, and permissions assigned 
to each admin for their assigned net¬ 
works. One other critical tool that 
helped ensure the accuracy of the 
host table was an IP discovery pro¬ 
gram that regularly checked each address to see whether it was in 
use. and by what type of device. 

This article focuses on the host table editor. Apart from the main 
requirement of solving the problems mentioned above, the users of 
HTF had their own ideas about how it would work, ideas that were 
often at odds with each other. Some wanted an interactive-style pro¬ 
gram, while others preferred it to be command-line driven so that 
scripts could be written around it. Therefore, 1 made the interface 
work both ways. The same hierarchical command structure can be 
called from the command line to make scripting easy. 

Introducing HTE 

Host I able Editor (HTE), available for download at 
http://www.samag.conn/code/, is a Perl program that manages a 
database of DNS information. See the sidebar for a summary of 
HTE tiles available with the code. The HTE system provides a uni- 
torm interface for entering DNS data and generating DNS-related 
configuration files. 

The advantages of storing Hostname, IP address. DNS. and 
DHCP information in a database system such as HTE include: 


Centralized control — It allows administrators to keep track of all 
host, IP address. DNS. and DHCP information in one common 
database. 

Security — By using a database, remote admins can access the 
host table data without requiring root access to the master servers 
that host DNS and/or DHCP maps. 

Error checking — Data input rules and automated building of host 
table, DHCP, and DNS configuration files helps ensure syntactic 
accuracy. 

Uniform updates — By using a single system and database, the 

data and the format of the automati¬ 
cally created files will remain neat 
and consistent. 

* Single source (only have to enter 
the data once) — This eliminates 
the need to manage separate host 
table. DNS. and DHCP configura- 
tion files, which can often reside 
on multiple servers. 

• Automatic file generation — In a 
shop where several servers are 
used to manage the network, 
changes will he propagated prop¬ 
erly without the need for manual 
intervention. 

Some possible disadvantages to this 
HTE program may have to do with 
unique or special needs of your 
environment. For example, the configuration files that are built were 
designed for use with standard DNS or DHCP servers on L'nix- 
based systems. Where third party server software is used, these files 
may not work, or there may be a number of required options that are 
not supported. 

This article describes how' to set up HTE. how to enter DNS 
information into the HTE database, and how to use HTE to auto¬ 
matically generate host files. DNS zone files, and DHCP configura¬ 
tion files. 

Installing HTE 

You will need a current version of MySQL and Perl. If you plan 
to run a DNS server, HTE was built with Bind 9.2.1. however, 
newer and older versions may work just as well. (Keeping Bind up- 
to-date is recommended.) Install the following Perl modules from 
CPAN: Term::Readline::Gnu. DB1, DBD::MySQL. Getopt::Long, 
Config::Simple. Data:: Dumper, and Net::Netmask. The 
Net::Netmask module (by David Muir Sharnoff) provides HTE the 
ability to keep your network structured. You may also want to install 
MySQECC to simplify database administration. 
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Unpack the HTE kit in a temporary directory and install it with 
the follow ing commands: 

perl Makefile.PL 
make 

make test 
make install 

Create a MySQL database to store the host table data using a suit¬ 
able name such as "hosts'’. Then, while still in the directory into 
which you unpacked the HTE kit. run the following command to 
initialize the database with a MySQL account that has create privi¬ 
leges: 

mysql -u admin -p hosts < hte.sql 

You then must create user accounts in MySQL. Begin with an 
account with a name like "build” with SELECT only privileges. 
This account will be used to create the hosts table, DNS, and DHCP 
tiles. This may also be used to give average users read-only access 
to the HTE editor. 

Copy hie.coni to /etc/hte.conf and put the values for your data¬ 
base and read-only user on the dsn, user, and pass lines. 

Much of the security of the system is managed by the database. 
If you want to specify sets of users who can create networks and 
those who add and delete host table entries, provide different per¬ 
missions in your grant statements. For example, a MySQL account 
for Super_Admin can be created with SELECT. INSERT. UPDATE, 
and DELETE on all tables in the hosts database. 

Super_Admin can create network definitions, DNS zones, and 
host table entries. Joe_Admin, who is only allowed to create host 
table entries, should have SELECT on all tables, and INSERT. 
UPDATE, and DELETE on just the hosts table. Network_Guy will 
have INSERT. UPDATE, and DELETE permissions on hosts and 
networks. You may not want Joe_Admin to delete addresses once 
they are created, so only give INSERT and UPDATE permissions. 
The editperms and glue tables are reserved to manage HTE and 
should be read-only for all users. The htevars table is where user- 


About the Files 


Here is a description of the files included with the 1.1 

Version of The Host Table Editor: 

Changes — This text file lists the changes made to each version 
of the package. 

hte — The host table editor program. 

hte.conf — A sample configuration file. 

hte.sql — The MySQL database creation script with initial data. 

MANIFEST — The list of files included in the package. 

Makefile.PL — The Perl makefile creation script. 

test.pl — Basic Perl test script used during installation. 

Commands/Commands.pm — The interactive command tree 
module. This defines the language and structure of the user 
interface. 

Commands/test.pi — Basic Perl test script used during installation. 

Commands/Makefile,PL — The Perl makefile creation script, 
used during installation. 

Commands/CmdTree.pm — The Perl object class that manages 
the command tree structure. 


defined preferences are stored. If you want to allow your users to 
save their preferences, make sure they can INSERT. UPDATE, and 
DELETE. Thus, the permission schemes can be provisioned in the 
manner most appropriate for your shop. 

To simplify using HTE. add the HTE USER environment variable 
to your .eshre or ,bash_profile. You may also want to set HTE_PASS. 
though it's not recommended if security is a concern. HTE will prompt 
you for a password if HTE_USER is set and HTE_PASS isn't. 

Take a Look Around 

Get familiar with the interface. The command levels can be 
stepped into one at a time, or chained together to jump to the desired 
level or invoke a desired command. For example, if you type 
"show” by itself, you will be at the show prompt and can then type 
any sub-level command. 

bash $> hte 
hte > show 

show > ip 192.168.254.12 
or: 

bash $> hte 

hte > show ip 192.168.254.12 
or, from the command line: 
bash $> hte show ip 192.168.254.12 

When you use commands from the command line, interactive mode 
is disabled. The only exception is when you create something, you 
will be placed into the entry editor to modify the values of the newly 
created item, You can also skip tiic editor by adding a set value 
command. For example: 

bash $> hte create zone fred.com set nameserver ns.myisp.com 

Once created, editing the zone from the command line is done the 
same way: 

bash $> hte edit zone fred.com set nameserver ns.myisp.com 

Typing help at any prompt will show the entire command tree from 
that level down. The command-line version is - - he 1 p. As you can 
see, the interface is quite flexible. 

Building Your Network 

If you are going to use DNS. you will want to stall by creating 
your DNS zones. Create your zone records and make sure you set 
the nameserver and SOA fields to the right values for vour setup: 

bash $> hte create zone fred.com set nameserver ns.myisD.com 
bash $> hte edit zone fred.com set SOA master.fred.com 

Now create at least one network to get started. The networks are 
created using the Cl DR format, which is common. When you edit or 
show a network, leave off the /bits value at the end: 

bash t> hte 

hte $> create network 192.168.101.0/24 
Network created, 
description : 
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network_address: 192.168.101.0 
bits : 24 

create_date : 20030531204143 

creator : dspeers 

dns_id : [] 

group_id : 

location : 

edit network) 

Type hel p to see what you can do in the edit mode and to see which 
fields are read-only. 

You will first describe your new network to make it different from 
all the other networks. The set command takes the field name vou 
want to set as its first argument. Everything that follows is the value: 


In the above example, "fred" was not a valid value for the dns_id 
field. In such a case, the program will use that invalid value as 
search criteria to find possible matches in the zone table. Each 
matching item is listed by a number that can be used to set the 
dns_id field. In this example, by setting dns_id to 1, the fred.com 
zone was assigned. It is also valid to set the name to an exact match 
in the zone table: 

edit network) set dns_id fred.com 

Once you have made your changes, press "Enter” once to leave this 
level. The program will notice that you made changes and ask you 
whether you want to save them. Congratulations! You have created 
your first network. Look at your new network: 


set description My own private network 

Typing sh at the edit network) prompt will redisplay the record 
w ith your new' description. 

Next, place this network in the DNS zone you created earlier. The 
DNS zone is assigned using the dns_id field. Notice how the dns_id 
field has square brackets in it, because it is a lookup field. If you try to 
assign any value that is not a valid key or a complete value in the lookup 
table, the possible values w ill be listed, using your argument as a filter: 

edit network) set dns_id fred 

The following are valid for dns_id (Filter: fred) 

1 => fred,com 

edit network) set dns_id 1 


hte ) show network 192.168.101.0 
description => My own private network 
network address => 192.168.101.0 


bits 

create.date 

creator 

dn$_id 

location 


-) 24 

=> 20030604135652 
=) dspeers 
=> [fred.com] 

=> 


Create a Host Table Entry 

Creating a host table entry is done in the same way as creating a 
network, which must already exist. HTE will already know to which 
network the address belongs. 
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hte > create ip 192.168.101,10 


Address created. 



description 

=> 


hostjiame 

«> 


ip.address 

=> 

192.168.101.10 

aliases 

=> 


create_date 

=> 

20030531220751 

creator 

=> dspeers 

device_id 

=> 


lastjeen 

=> 


st atus_id 

edit host) 

=> 

[assigned] 


The only other field that must be set is the host_name. The status_id 
field is used to help provide guidance to other admins on which 
addresses are in use. available, or reserved for future use. 

Building the Host Table 

Building a host table is as easy as typing "build host table". The 
table will be grouped by network to keep it looking neat: 

# 

# Network : 192.168.101.0/24 Mask: 255.255.255.0 

# Location : My House Domain: fred.com 

# Description: Middle Earth 

# 

192.168.101.10 bilbo # Dan’s server 

192.168.101.11 frodo if File Server 

192.168.101.12 epson if My Printer 

You can capture the output in traditional Unix fashion by redirecting 
the output to /etc/hosts, or by setting the output variable: 

hte> build host 

host > set output /etc/hosts 

Building the DNS Maps 

The host table editor will create both the named.conf file and 
the individual zone files. Often the named.conf file will need 
more options than the Host Table Editor will support. Your shop 
may also have things that you don't want the Host Table Editor to 
manage for you. or other configuration settings you want to set. 
The best practice is put your standard options in /etc/named.conf 
and use an include file command to reference the Host Table 
Editor portions. To do this, you have the Host Table Editor output 
its master DNS configuration data to a file named 
/etc/named.auto, which would look something like this in your 
/etc/named.conf file: 

options { 

directory "/var/named"; 

} 

include "/etc/named.auto''; 

The default settings for the build dns maps command will create the 
/etc/named.auto file and will put the zone files in the /var/named direc¬ 
tory. You must add the include statement to your /etc/named.conf file. 

The settings that determine where the files are saved can be 
changed in interactive mode at the blii Id dns level: 


hte > build dns 
dns > vars 

config => /etc/named.auto 

dir => /var/named 

verify -> /usr/sbin/named-checkzone 

dns > set dir /var/named.d 

If you have a current version of Bind installed, it comes with a 
helper program to verify the format of the zone files. It will run as 
each zone file is created. 

Here is a practical example of how all these pieces of DNS will 
fit together. Say you had a DNS domain called "myplace.com" and 
in that network is a machine called foobar with the IP address of 
192.168.200.37. Your DNS server will live on a box called 

ns.myplace.com and have an address of 192.168.200.5. 

To begin, create your DNS zone: 

hte > create zone myplace.com set nameserver ns.myplace.com 

hte > edit zone myplace.com set SOA ns.myplace.com 

Then, define the network where your hosts will live: 

hte > create network 192.168.200.0/24 set dns_id myplace.com 

Next, add the hostnames to the network: 

hte > create ip 192.168.200.5 set host_name ns 

hte > create ip 192.168.200.37 set hostjiame foobar 

There may also be extra records you need in a zone file. For exam¬ 
ple, you may want to define an alias for foobar and have it also 
known as www.myplace.com. You may also want to define how 
email gets to your mailserver. These are accomplished with the cre¬ 
ate zone_record command. It takes no arguments and will allow a 
variety of record types, including MX records to extra CNAME 
entries. A typical addition of an MX record to handle mail will look 
like this: 

hte > create zone_record 
Zone entry created. 


address 

*> 

dnsjd 

=> [] 

entryjd 

=> 3 

priority 

-> 0 

record 

=) IN 

target 

-) 

type 

-> [] 


edit zone_entries> set address myplace.com. 
edit zone_entries> set dn$_id myp1ace.com 
edit zone_entries> set target foobar 
edit zone entries) set priority 10 
edit zone_entries> set type MX 

This MX record will allow mail bound for myplace.com to be han¬ 
dled by foobar.myplace.com. Note the period at the end of the "set 
address" line. Anytime you set the address or target fields to a fully 
qualified domain name instead of just a hostname, you must include 
the period to make sure the nameserver doesn t automatically add 
the local domain name for you. 

To make people think that foobar also goes by the name of 
www.myplace.com, you could create another zone record using the 
CNAME type using these settings: 
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edit zone_entries> set address www 
edit zone_entries> set dnsjd myp1ace.com 
edit zone_entries> set target foobar 
edit zone_entries> set type CNAME 

And the last step will create the DNS maps; 

hte > build dns maps 

The named.auto tile will look something like this: 


zone "myplace.com"! 
type master; 

file "myplace.com.zone"; 

}; 

zone "254.168.200.in-addr.arpa"! 
type master; 

file "254,168.200.in-addr.arpa.zone"; 
1 ; 


The myplace.com zone tde may look a bit cryptic, which is why 
you wanted a program like the Host Table Editor to make it for you: 


;Zone built by HTE 
ITTL 86400 

@ IN SOA ns.myplace.com root.local host ( 

11 ; serial 
28800 ; refresh 
7200 ; retry 
604800 ; expire 
86400 ; ttl 

) 


e 

IN 

myplace.com 

IN 

WWW 

IN 

ns 

IN 

foobar 

IN 


NS ns.myplace.com 

MX 10 foobar 

CNAME foobar 
A 192.168.200.5 

A 192.168.200.37 


Building DHCP Config Files 

DHCP has become a required service as the popularity of wire¬ 
less networks increases. Automatically creating the configuration 
for DHCP is a natural extension of the Host Table Editor. In just a 
couple of easy steps, your DHCP server will be ready to go. 

To keep things simple, a DHCP entry is associated with a net¬ 
work. The network records already know things like the net mask 
and DNS domain name, so the only required step is to define the 
range of IP addresses that will be available for DHCP clients to bor¬ 
row (or lease in DHCP terms), which is done by giving the starting 
and ending address range for the desired network. 


hte > edit dhcp 192.168.254.0 set range 192.158.254,100 192.168.254,120 


DHCP as a protocol supports a number of optional convenience fea¬ 
tures to help the weary transient user navigate around your network. 
These include things like the address of the router to use to get off 
the local network, and address of your DNS server: 


hte > edit dhcp 192.168.254.0 set router 192.168.254.1 

Many of the options you could provide depend largely on how your 
DHCP clients are configured and the services your network offers. 


For example, say your network has an NTP (Network Time 
Protocol) server to set your workstation clock against. You would 
set the ntp-servers option: 

hte > edit dhcp 192.168.254.0 set ntp-servers 192.168.254.1 

For a list of valid options, see the dhcpd-options manual page that 
comes with the ISC DHCP client or the IETF draft on DHCP agent 
options at http://www.ietf.org. 

Last, to build the dhcpd.auto configuration file, type “build 
dhcp" at (he hte> prompt. Here you can set the global options of 
max-lease-time and default-least-time by using the set command. 
The default values will be sufficient for most sites: 

hte > build dhcp conf 
Building dhcpd.auto file... 

’192.168.254.0/24’ 

Done. 

Here is w hat the resulting dhcpd.auto file will look like: 

subnet 192.168.254.0 netmask 255.255.255.0 { 

default-lease-time 21600; 

max-lease-time 43200: 

option subnet-mask 255.255.255.0; 

option domain-name "myplace.com"; 

range dynamic-bootp 192.168.254.10 192,168.254.20; 

option routers 192.168.254.1; 

} 


You will use an include statement in /etc/dhcpd. conf just same way 
we did for /etc/named.conf, which will set a variety of static and 
default options for your DHCP server. 

Summary and Future Features 

By now you have created DNS zones, networks, and host table 
entries. You now have effective ways to manage access to the host 
table, and automated ways to build your /etc/hosts file, DNS, and 
DHCP configuration files. This should dramatically simplify the 
way you manage your network addresses, improve security, and 
will keep all addresses and related configuration files in sync. 

At this point, will want to create a crontab entry to build the var¬ 
ious configuration files on a regular basis. An example contrab 
entry looks like this: 

0 * * * * /usr/bin/hte build host table;/usr/bin/!ite build \ 
dns maps:/usr/bin/hte build dhcp conf 

Some issues in how you manage your network remain, such as how 
to know when an address is no longer in use. or whether one has 
been taken without being allocated in the database. You may also 
want to know what kind of devices are using those addresses. These 
features are planned for in an add-on program called ip-discover 
that may be available by the time you read this. In the original sys¬ 
tem. ip-discover used both iemp requests and SNMP gets to deter¬ 
mine which addresses were in use and what was using them. 

Dan Speers has been in the technology industry for 17 years and is currently 
a Professional Services Engineer with VA Software. He has been a Linux 
user since kerne! 0.98. ,4? every opportunity, he helps promote the use of 
open source software in roles traditionally filled by commercial products. 
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